Wikimedia Apps/Team/Wikipedia Year in Review/2025 Year in Review/Personalization for logged-out users experiment
Summary
[edit]This page details an A/B test experiment that the iOS app will be running alongside the Year in Review feature. We'll test allowing users to view personalized reading insights without logging-in. This A/B test aims to understand the relationship between personalization and logged-in reader retention and multi-day engagement, contributing to the efforts of the 2025-2026 annual plan where we engage core consumer audiences.
Background
[edit]As part of the annual plan, we are testing improvements to Year in Review, to learn if it results in an increase in overall app retention for logged-in users. We ran user testing with the iOS version of Year in Review, and saw that users overwhelmingly preferred the personalized slides over the collective slides.
60% of our free-text survey responses from last year's edition of Year in Review were also requesting more personalization. Year in Review currently requires an account to view personalized reading insights. However since the insights are stored on device, we have the ability to show them to logged-out users as well. We want understand if personalization alone increases retention as much as creating an account to view personalization.
Experiment Requirements
[edit]- 50% of those who open Year in Review & click "Get started" are sorted into control, 50% into variant
- Control (no change from current behavior)
- Users must be logged-in to be eligible for personalized reading slides
- Show log-in prompt when someone clicks "Get started"
- Show collective slides when someone is logged-out
- Show collective slides when someone does not have any data
- Variant B
- Users do not need to be logged-in to see personalized reading slides
- Do not show log-in prompt when someone clicks "Get started"
How will we measure success
[edit]Validation
[edit]- KR 5.1 - 1% increase in overall app retention among readers who were required to have an account (A) compared to those who did not need to have an account (B).
- KR 5.2 - 10% increase in feature return rate for logged-out users who were shown personalized slides vs those who were not.
Guardrails
[edit]- GR 5.3 - No more than 10% of all survey responses from logged-out users express concerns with privacy
Curiosities
[edit]- CR 5.4 - How does the completion rate, satisfaction rate and donation rate for logged-out users in group A compare to group B?
How does this work fit into the Wikimedia Foundation's Annual Plan?
[edit]Under the Wikimedia Foundation's Infrastructure Goal, and within the group of objectives focused on Wiki Experiences, is an objective related to improving the experience of consumers:
Wiki Experiences 3: Consumer experience Objective: Readers from multiple generations engage, and stay engaged, with Wikipedia, leading to measurable increases in retention and donation activity
Several Wikimedia Foundation teams are committed to working on projects under the WE3.3 Key Result: Draft Hypotheses.
Wiki Experiences 3.3 Key Result: Demonstrate a practically significant increase in logged-in reader retention, as measured through A/B testing of one feature per platform, by the end of Q2 FY2025-26.
| Hypothesis | Timeline | Phabricator epic | Status |
|---|---|---|---|
| Wiki Experiences 3.3.10
If we A/B test requiring an account to view the personalized reading insights of Year in Review, we’ll see a 1% increase in overall retention rate for users that were required to have an account, compared to those that were not. |
11 October 2025 – 31 December 2025 | T382442 | In progress |
Results
[edit]- Key results
- Goal: 1% increase in overall app retention among readers who were required to have an account (A) compared to those who did not need to have an account (B).
- Actual: 2% decrease in overall app retention among readers who were required to have an account (A) compared to those who did not need to have an account (B).
- In comparing segments of users, we saw that logged-in users still had the highest retention rates with 86% retained, compared to logged-out users with access to personalization at 79% retained.
- Goal: 10% increase in overall app retention users who were shown personalized slides vs those who were not
- Actual: 7% increase in overall app retention from users who were shown the personalized experience compared to non-personalized users
- Logged-out users with access to personalization retained at 3% higher rate compared to those who did not have access to personalization. This learning was statistically significant.
- Goal: No more than 10% of all survey responses from logged-out users in Group B express concerns with privacy
- Actual: Only 3% of logged-out users who answered the survey expressed concerns about data tracking or privacy.
- Goal: 1% increase in overall app retention among readers who were required to have an account (A) compared to those who did not need to have an account (B).
- Learnings
- While requiring an account leads to the highest retention rate, there are major benefits to removing this requirement and allowing logged-out users to see personalization: it significantly increases logged-out reader retention, satisfaction, and donation rate. Because logged-out users currently make up ~90-95% of our users, this has a large impact on our overall results.
- The satisfaction rate for logged-out users with access to personalization was 3% higher (97%) than logged-out users without access (94%).
- Logged-out users in Group B (no account creation prompt, or account creation requirement) had a 53% higher completion rate (48% completion) compared with Group A (31% completion).
- The donation rate for logged-out users in Group B was 327% higher than in Group A. Group B had a 2% donation rate, and Group A had a 0.5% completion rate.
- Viewing personalized insights through Year in Review was still a strong motivator to create an account and an effective way to increase our share of logged-in users: In Group A with the account requirement, 12% of Year in Review users in control were logged in, compared with only 8% in the variant where an account wasn’t required, a 50% increase in logged-in sessions for Group A.
- If we increase our share of logged-in users, we can increase overall retention to the app: we saw that logged-in users still had the highest retention rates with 86% retained, compared to logged-out users with access to personalization at 79% retained.
- While requiring an account leads to the highest retention rate, there are major benefits to removing this requirement and allowing logged-out users to see personalization: it significantly increases logged-out reader retention, satisfaction, and donation rate. Because logged-out users currently make up ~90-95% of our users, this has a large impact on our overall results.
- Next steps
- When planning next year’s Year in Review, these results will allow us to make decisions about the tradeoffs between the benefits of motivating account creation motivation in the long term, and yearly performance benefits of showing personalized insights to logged-out users.
- We will consider allowing all logged-out users to see a personalized version of Year in Review, moving the account creation prompt to the end of the flow, and motivating users to create an account to receive extra insights, or an extra reward.
How to follow along
[edit]Our work is being coordinated on this phabricator epic: T382442. Updates will appear on the Updates page each month. At the end of the experiment, we will share results and next steps on this project page.
Feedback? Leave a comment on the project discussion page—your insights shape this experience!