Jump to content

Readers/Listening to Wikipedia/Phase 1

From mediawiki.org


Background

[edit]

Giving app readers more flexibility and variety in how they consume Wikipedia content is expected to deepen engagement and bring them back more often, ideally daily. The Explore feed redesign, which added a personalized tab in a more modern format, already lifted Home feed retention by 10.5%. This experiment tests whether adding a listening format into the "For you" feed can deliver further improvements in reader retention.

Supporting text-to-speech across the full Wikipedia corpus and all languages requires infrastructure investment that hasn't been made yet. Before committing to that, this MVP is testing a narrower question: Does text-to-speech content change app retention at all? To do that, the team is using a fixed, pre-generated corpus of articles.

Why a text-to-speech model vs on-device voice

[edit]

User research found that people overwhelmingly prefer a natural, human-sounding voice over the robotic, on-device voice options already available (like Android's built-in "read aloud"). Testers said the on-device voice sounded "too robotic" and that they wouldn't use it for extended listening. That pushed the teams toward testing with an improved text-to-speech model. Learn more about why the Machine Learning team chose Kokoro and how the team is thinking about model choice longer-term.

Scope of Experiment

[edit]

The "For you" feed on the Apps already recommends content to users based on topics they follow, designed to be a low-effort, scroll-and-sample surface. Suggesting a 1-2 minute second article section for listening fits that existing behavior; a full-article listen is a different mode entirely, closer to podcast or audiobook use. Testing short-form first also lets the Mobile Apps team learn whether text-to-speech content changes retention at all, before taking on harder infrastructure problems like real-time generation, full corpus coverage, or multilingual support.

User story

[edit]

As a Wikipedia app user, I want to listen to snippets from articles while I'm scrolling "For you" in spare, in-between moments, so I can absorb something interesting without needing to stop and read.

Experiment set-up

  • This experiment will start as a time-bound A/B test for 30 days before being removed from the app. Next steps will be determined only once results are reviewed & shared.
  • 50% of users see the existing "For you" feed (control); 50% are eligible for the listen module (variant B).
  • Only around 3% of app users currently have topic interests selected, so only ~3% of users in group B will be eligible for the listening experience.

Feature requirements

[edit]
  • Where it shows up: The listen module sits at the top of the "Based on your interests" section in "For you," and only appears for users who have Topics selected and English set as their feed language
  • Introducing it: A one-time tooltip shows first app open after launch, shown only to users with at least one Topic selected. Can be dismissed
  • Carousel of content: Module contains 4 cards, each tied to a different article from a topic the user follows, refreshed daily from a fixed, quality-vetted corpus of about 8,000 English articles. Playback stops if a user swipes to another card.
  • What a card includes: Article title, a preview of the first section, snippet length, and a play button. Tapping the title or image opens the full article; tapping play reads the first section aloud with word-level highlighting synced to the audio.
    • Attribution: The feature communicates to users that the content was generated using a text-to-speech model, so they do not mistake it for a human voice
    • Signaling it's an experiment: the feature is labeled "Beta," with an info menu linking to this project page
    • Date generated: The feature will include the date the audio was generated, if the revision ID on the audio is no longer matching the current revision ID on the article
    • Ability to report issues: The feature will allow users to "report an issue" by opening a pre-filled support email
  • Users can opt out: Any user can use the three-dot menu to turn off the module and re-enable it anytime from settings
  • Learning what people want: A one-time survey (shown after a user's second play, or 7 days out for users who haven't returned) asks whether they'd want more of an article, full articles, or excerpts across a topic.

Not in MVP scope: An entry point from within the article view, full save-and-organize for snippets, skip/adjustable playback speed, multiple voice options, multilingual support, offline download, or expanding beyond the fixed article corpus.

Prototype Video

[edit]

How we'll measure success

[edit]

Hypothesis:

[edit]
  • If we add a short-form text-to-speech experience (the first article section) into the "For you" feed in the Android App, we’ll see 10% of users who interact with the feature once (press play on an audio file) return to interact again on a second day.

Key results

[edit]
  • Stickiness: 10% of users who interact with the feature once (press play on an audio file) return to interact again on a second day (Baseline: 11% return rate for based on your interest impressions)
  • 15% of users who have the feature available to them engage with it (feed module impression)

Guardrails

[edit]
  • Overall app 2nd day retention is flat for variant group compared to control group.
  • Less than 20% of feedback received mentions the voice quality being robotic or non-human

Curiosities/Considerations

[edit]
  • Survey results: What share of users are interested in listening to more sections, the full article, or excerpts from different articles. What are the main requests and feedback among the freetext? Are there differences in results for retained vs non-retained users?
  • Listen length: What is the median cumulative listen time per session?
  • Funnel: What share of users who had an impression of the module tapped play? What share of users listened to multiple articles within the carousel? [Support email]: What are the most common issues reported by users?
  • Feed retention: Did "For you" feed retention differ for control group vs variant group?
  • Engagement comparisons: How does the engagement with the "Listen to article intros" module compare with the other "For you" modules? What share of unique users interact with each one.

How to follow along

[edit]

We’ll add regular updates to the updates tab. See the full product requirements and follow-along with our work on the Phabricator Epic: T434720. Leave us feedback or suggestions on the Discussion page.

Experiment FAQ

[edit]

Can I listen to any article?

[edit]

Not during this experiment. We’re using a fixed collection of selected English articles with audio generated in advance. The listening cards suggest articles based on the topics you follow.

Can I listen to full articles?

[edit]

This experiment focuses on article introductions. Full-article listening is not included, although it is a feature we might explore in the future. Through the survey, you can tell us whether you’d prefer introductions, more sections, full articles, or short excerpts from different articles on a topic.

Does the audio update when an article changes?

[edit]

Not automatically for this experiment. The recordings are prepared in advance, so they may reflect an earlier version of an article.

If an article has changed since the version used for the recording, the feature shows the date of the article content used. You can open the article to read its latest version.

What if the voice mispronounces something or the audio sounds wrong?

[edit]

Please report it through the feature’s “Report an issue” option, which opens a support email. Include the article title and the word or passage where you noticed the problem. Reports about pronunciation, missing words, or playback problems help us understand what needs improvement.

Can I download the audio, change the voice, or adjust the speed?

[edit]

These options aren’t included in this experiment. Saving an article saves the article, not its audio.

How long will the experiment last?

[edit]

The experiment is planned to run for 30 days. Afterward, the feature will be removed while the team reviews the results and decides on next steps.

What will guide the decision about what happens next?

[edit]

We’ll look at whether people return to the feed and listen again, what they tell us in the survey, and the issues they report. Community feedback about the voice, pronunciation, and audio quality will also inform next steps.

Interest in listening would support exploring the idea further. It would not, on its own, settle which model, voice, or approach to use.

Does this experiment mean you’ve decided to use this model and voice long-term?

[edit]

No. Testing this model and voice, including in the production version of the app, does not mean we’ve chosen them for a permanent feature. Any longer-term decision would need to consider reader interest, audio quality, community feedback, and what we learn from the test.

Is AI writing or changing the articles?

[edit]

No. The feature uses a text-to-speech model to turn existing Wikipedia text into audio. It does not rewrite, summarize, or add content to articles. Learn more about the model currently being used.

The words come from Wikipedia articles written and edited by contributors.

Is the voice a person’s recording?

[edit]

No. The audio is generated using a text-to-speech model called Kokoro. It can sound like a person speaking, but it is a synthetic voice. The feature includes a notice explaining how the audio was made.

Why use this model instead of the phone’s built-in voice?

[edit]

In our usability testing, participants preferred a natural-sounding voice over the on-device voices tested. Some described the on-device voices as too robotic for longer listening. This experiment will help us understand whether a more natural voice makes listening an experience readers want to use regularly.

The team will be keeping an eye on developments within the on-device voice models and viability for using them instead.

Where can I share feedback or follow progress?

[edit]

You can report problems through the in-app “Report an issue” option and share broader feedback on the project’s discussion page.

Visit the experiment page for details and the updates page for progress and findings.