Reading/Web/Projects/Related pages/Moving to stable

Plan and analysis
The Reading team would like to move forward with making related pages a stable feature on mobile web on all Wikipedias (except for the German Wikipedia, which rejected the feature, unless they decided to re-think their vote). The decisions is based on the fact that feature performance on mobile after over 3 months of running in beta (+ 20% click-through rate) is enough matrix to enable us to move the feature to stable. The click through rate is an indication that readers are inclined to check further related topics on the articles as we offer them. This is a good starting point, in which further feature development can stem from.

Timeline and tasks

 * Announce the plan and time line by Sept 1st
 * Allow 3-4 weeks of discussion
 * On Desktop: Remove Related pages from desktop beta feature on all Wikipedias --if a community needs an exception, let us know; we understand that the feature might be helpful in populating article in relatively small or new Wikipedias
 * On mobile: Start a staged roll out, where we exclude the top 6 Wikipedias in terms of article count, deploy the feature to all other Wikis at once. This helps making stable bug fixes easier, given smaller traffic
 * If all goes well, start deployment according to the above plan by early October.

Community feedback summary

 * Concerns/comments around how the algorithm behaves
 * Changes have been deployed to the way how the logic of the selected articles is implemented. Initially, 2/3 of the criteria for related pages were essentially popularity (links in and featured article status). Only 1/3 was similarity.  We now moved it to being 100% about similarity.(Details)


 * Concerns around the display non-free images on English WP
 * This has been resolved via Extension:PageImages fixes and will continue to have feature enhancement


 * Concerns around the click-through rate as a sole metric for measuring feature success.
 * As mentioned above, the click through rate is a helpful indication to help us realize user interest in the idea of offering articles related to the topic they are reading/browsing.


 * Related pages and See also sections
 * Currently, editors can override the suggested articles by placing articles manually. In the future we will be looking into different patterns of how see else is significant/present in each Wikipedia, and adapt Related Pages to be as beneficial as possible to each Wikipedia

Summary
Clickthrough rates for related pages links are especially high (20.2%) when users see the related pages section at the bottom of the article. The overall clickthrough rate on related pages links for mobile is 7.5%. Rates are significantly (5.4 times) lower for desktop (1.2% and 2.8%, respectively). No significant changes in these rates were observed over time, indicating that users will continue using related pages with a similar frequency.

Details
On mobile, the clickthrough rate for any related link given is 0.0748 clicks per page (ranging between 0.050 and 0.099 with 95% confidence). When users view the related pages (by scrolling to the bottom of the article), the clickthrough rate is 0.2019 clicks per page with related pages seen (range 0.146 - 0.258 with 95% confidence). No significant changes in this trend were observed over time.

On desktop, the clickthrough rate for any related link given is 0.0117 clicks per page (+/- 0.0066, 95% confidence). When users view the related pages (by scrolling to the bottom of the article) the clickthrough rate is 0.0278 per page with related pages seen (+/- 0.0168, 95% confidence). No significant changes in this trend were observed over time.

Query:

SELECT count(*) AS num_Events, event_eventName,

LEFT(TIMESTAMP, 8) AS dt, event_skin

FROM RelatedArticles_14496900

GROUP BY dt, event_eventName, event_skin;