Udemy’s Learner streak experiment
The problem
- Retention is significantly lower for smaller customers who don’t have dedicated Customer Success Managers.
- Driving engagement via product may help retention for these customers since organic learner engagement is strongly correlated with churn: the more learners are engaged, the least likely the customer is to churn.
How might we improve learning engagement via product?
Ideation workshop
Participants:
- Team (PM, Design and 3 Engineers), Customer Success and Learning & Development
- Intro: Data and research overview
Brainstorming:
- What are the possible causes for the behaviour we see in this data?
- Rewrite problems as 'How might we'
- Ideation on HMWs
Storyboard ideation
Participants:
- Pod (PM, Design and Engineers)
Brainstorming:
- How would most voted ideas look like in product?
- Diverge > Discuss > Converge
- Final storyboard was presented to stakeholders for feedback before design explorations.
.jpg)
.jpg)
Scoping workshop
Goal:
- Decide on MVP we can ship as an experiment so we can test our hypothesis.
Workshop activities:
- What do we know?
- What do we want to learn?
- How can we learn what we want?
- Define phases based on prioritisation
Hypothesis
Learners who have a time-based goal will be more engaged with learning
Multiple quick iterations to explore layout and data visualisation
Design reviews to ensure product consistency
Feedback from stakeholders
.jpg)
Validation
To quickly validate the designs before launching the experiment we ran an unmoderated user test and tested two options.
Research questions:
- Can users quickly understand the charts?
- Would they find it motivating to have goals and streaks?
Results:
- Users could quickly understand both options but the donut bar performed better.
- Users found the visualisation motivating and engaging.
.jpg)
.jpg)
