Udemy’s Learner streak experiment
The problem
- Retention is significantly lower for smaller customers who don’t have dedicated Customer Success Managers.
- Driving engagement via product may help retention for these customers since organic learner engagement is strongly correlated with churn: the more learners are engaged, the least likely the customer is to churn.
How might we improve learning engagement via product?
![](https://cdn.prod.website-files.com/61eaa667e0452b8247b687b5/663a407d924b73bf75d21b65_Screenshot%202024-05-05%20at%2016.58%201.jpg)
Ideation workshop
Participants:
- Team (PM, Design and 3 Engineers), Customer Success and Learning & Development
- Intro: Data and research overview
Brainstorming:
- What are the possible causes for the behaviour we see in this data?
- Rewrite problems as 'How might we'
- Ideation on HMWs
Storyboard ideation
Participants:
- Pod (PM, Design and Engineers)
Brainstorming:
- How would most voted ideas look like in product?
- Diverge > Discuss > Converge
- Final storyboard was presented to stakeholders for feedback before design explorations.
![](https://cdn.prod.website-files.com/61eaa667e0452b8247b687b5/663a4077590a070af085eb3a_Screenshot%202024-04-19%20at%2012.30%201%20(1).jpg)
![](https://cdn.prod.website-files.com/61eaa667e0452b8247b687b5/663a4073e2317f1827d3ba54_Screenshot%202024-04-19%20at%2012.34%201%20(1).jpg)
Scoping workshop
Goal:
- Decide on MVP we can ship as an experiment so we can test our hypothesis.
Workshop activities:
- What do we know?
- What do we want to learn?
- How can we learn what we want?
- Define phases based on prioritisation
Hypothesis
Learners who have a time-based goal will be more engaged with learning
Multiple quick iterations to explore layout and data visualisation
Design reviews to ensure product consistency
Feedback from stakeholders
![](https://cdn.prod.website-files.com/61eaa667e0452b8247b687b5/663a40d5924b73bf75d26952_Screenshot%202024-04-19%20at%2016.50%201%20(1).jpg)
Validation
To quickly validate the designs before launching the experiment we ran an unmoderated user test and tested two options.
Research questions:
- Can users quickly understand the charts?
- Would they find it motivating to have goals and streaks?
Results:
- Users could quickly understand both options but the donut bar performed better.
- Users found the visualisation motivating and engaging.