Reading/Web/Projects/Related pages/Draft RFC

RFC: Related Pages extension
This is an RFC relating to the Related Pages extension. It is currently enabled on beta, for mobile version of Wikipedia. Editors may test it on desktop by activating it in the Beta section of Special:Preferences. The Wikimedia Foundation Reading team wants to know, given the feature performance in beta, what would be the best path to move forward, in the following aspects: What needs to further improve or change in the feature? What would be the best way to move forward with the feature? What is the best way to plan a partial roll out on some Wikipedias to continue testing the feature and measuring its performance, given that beta, so far is only allowed for logged in users on desktop?

Background
The Related Pages extension (also known as the Read more feature) is an experiment carried out by the Reading team in order to further engage readers and help them learn more. It is primarily geared towards users who are interested in learning, but do not have a specific goal in mind. A user reaching the bottom of an article is shown the title and lead image for three articles that relate to the article they just finished. The related articles are selected by software based on textual similarity to the current article, with a preference for articles with many inbound links, and a preference for articles carrying the templates for Featured article, Featured picture, Featured sound, Featured list, or Good article. (More details at Related_pages#FAQ.) Editors may override these software selections by adding up to three to the page.

The premise of Related Pages is that users who get to the bottom of an article have passed over blue links and "See also", have finished the article, and are still interested in reading more. If a reader has reached to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for. While Related pages is primarily intended to increase learning by offering suggestions to the reader, learning is a very hard value to measure. As a result, we have been looking at engagement with the feature. This has been released on apps and saw a 16% click-through (for users who saw it). On mobile web, there is a 19% click-through (for users who saw it) The desktop numbers are much lower at 3.4%. While this rate is far higher than the rate of any 3 links on the page (see: https://meta.wikimedia.org/wiki/Research:Improving_link_coverage) and certainly higher than those towards the bottom of the page (see: ) there is some concern that this is simply because there is an image. To counter that, there is data that suggests the new clicks are additive--meaning they lengthening reading sessions and are not taking clicks away from other links, but these results are not definitive.

Rationale: If readers are offered suggestions that are similar to the topic they are reading about, this will further engage them, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics.

Early Announcements: Related pages extension had a documentation published on mediawiki since mid October 2015. It was also announced in Reading team updates, published in early November 2015. It was announced on Wiktech-l. In addition, there was a post made on the English Wikipedia Village pump in mid November 2015, announcing the work on the feature in beta.


 * A brief summary of some of the initial feedback at the project talk page
 * The community was not consulted first and we would have helped you avoid major pitfalls.
 * Reading Team Response: There was no structured consultation in ideation phase, because we don't yet have clear mechanisms for doing so, however, there was an update around the team's thinking to start working on a "Read More" feature, similar to the one on the web, this was published in Reading team updates, published in early November 2015. In addition, there was a post made on the English Wikipedia Village pump in mid November 2015, in order to update the community about the work happening on the extension. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature and others, we hope that you can help us move forward together.


 * Some language Wikipedias, notably German and Russian, have policies prohibiting collections of related links unless it is an objective and complete set.
 * The Reading Team does not want to push the feature on wikis that don't want it.  The reading team is not looking forward to break any community policies, however, it is essential to understand how exactly does Related Pages conflict with Thermenringe policy on German Wikipedia. It also is important to understand the different angles of the discussion on Russian Wikipedia, if there was a discussion.


 * English Wikipedia policy on non-free images does not allow their reuse for decorative or navigational purposes.
 * Non-free images will be excluded.


 * This is the same as see-also and does not add additional value.
 * See also sections don't exist on all articles. The section has less presence on smaller Wikipedias. At this is a point of the discussion, and one aspects which this RFC can hep define, is how a feature that helps readers further learn about a specific topic by offering more articles related to the topic that they are reading about, can also help the editing community better shape our content in articles. The Reading Team believes the sustained high click through rate on mobile suggests that users are finding it valuable enough to justify the feature.


 * Wikipedia, unlike many commercial websites, is not about generating as many clicks as possible.
 * This is not about generating clicks. The click itself isn't of value. This is about availing more knowledge by offering more articles related to the topic that readers are learning about.  A good website is composed of interlinked pages, that are easy to discover and find, which is one reason why Wikipedia identifies an orphan page as problem that needs action.
 * Many Read more suggestions are redundant to links in the article, and other suggestions can range from suboptimal to bizarre or even damaging. Generic products tend to be given links to arbitrary brands, for example Read more at the Hard disk drive article suggests Seagate Technology and at Car it suggests Mercedes-Benz. Potentially offensive articles and images can show up unexpectedly, such as images of penises and maggot infested wounds. At Korur language it suggests Anus, Anal and Poop. At List of serial killers before 1900 it suggests Peru national football team. At Murder of Kelly Anne Bates it suggests Batman.
 * Reading Team Response: For 'damaging' results, which are rare, editors can over-ride results. For 'suboptimal results', for now I think we should live with it because the readers seem to be deriving value.


 * The software returns particularly poor results for Disambiguation pages, for example at Holocaust (disambiguation) it suggests Batman.
 * The Reading Team sees little utility for the feature on Disambiguation pages. The feature is being disabled on those pages.

Further Points

 * How could Related Pages help with orphan articles?
 * Can we use related pages to explore different editing modes on mobile?

Enable on Desktop

 * The reading team would propose that a low traffic a/b test be the first step here, to see if readers exposed to "related pages" end up spending more time on the site. Unfortunately, we cannot track return rate (a proxy for overall satisfaction) due to privacy concerns.  Another alternative would be to run a survey to see if users who click on a related page entry have any difference in satisfaction over other users, but these self-reported measures of satisfaction rarely change (i.e. the average is ~4.4 stars rarely wobbles)

Enable Mobile only

 * On mobile we could so something like the above, or simply roll it out on several wikis at a time. The argument for the latter is that this has been on the apps for almost a year and seems to be doing well.