Reading/Web/Projects/Mobile Page Issues/AB tests

From September to November 2018, we ran A/B tests on the Latvian, Persian, Russian, Japanese, and English Wikipedias. Our goal was to determine the performance of improvements we made to templates that show messages regarding page content (page issues). Previously, the content of these templates was only accessible through a small link at the top of the article, making it difficult for readers and editors to know when the articles they are reading on the mobile website have content issues. Through this work, we made the templates more visible by displaying parts or all of the template text on top of the article. We also wanted to note the severity of each message, which we attempted to achieve by displaying each template with the severity and color it has on the desktop version of the template. Our objective for the project was to increase awareness of particular issues within an article on the mobile web. Through this A/B test, we wanted to gain insight on whether this goal was achieved by looking at the rates of interaction (clickthrough) of our new treatment versus the original treatment. We also wanted to gauge the effectiveness of our severity level variants by comparing the clickthrough for issues of different severity levels. Finally, we wanted to assess whether these changes had any immediate effects on the rates of mobile editing. Our hypotheses were the following:


 * The clickthrough rate for the new treatment will be higher than the old treatment
 * The clickthrough rate for issues of higher severity will be significantly larger than the clickthrough rate for issues with lower severity
 * Articles with the new treatment will be edited more frequently than articles with the old treatment

Our results displayed that two of the above hypotheses were correct: users interact with the new treatment more frequently than the old treatment and when they do, they tend to interact with higher-severity issues at a higher rate than lower-severity issues. However, our results showed that this increased engagement did not lead to any changes in edit rates. Our conclusion is that this feature improves awareness of issues with articles on the mobile website. Our results also suggest that readers understand the severity of these issues correctly and pay more attention to issues with greater severity.

Does the new treatment increase the awareness among readers of page issues?
The clickthrough ratio (for top-of-page issues notices) increased markedly with the new treatment on all five wikis (e.g. over 7x on ruwiki). We can confidently assume that the new design increases the awareness of page issues among readers. ... (table and charts go here)

... ..
 * Is there an increase in clickthrough based on the new issue treatments (from the article page to the issues modal, from the issues modal to anywhere else - details about issues type, modal dismissed, etc, i.e. where do people go after the modal)?
 * Does clickthrough depend on the severity of each issue?

Do mobile edits increase with page issues as referrer?
For practical reasons, we limited ourselves to measuring taps on the edit button (rather than actual saved edits). (table and charts go here)

Based on the data we can reject the hypothesis that those edit button clicks would increase (on pages with issues) in the new design. To the contrary, we even saw a slight but statistically significant drop in edit button clickthroughs on four of the five wikis. Based on our current understanding, we don't yet regard this as evidence of a detrimental effect of the new design, e.g. considering thee absence of a clear explanation of a mechanism that could cause this (keeping in mind that what we could measure here are only taps on the button, not finished edits, so the observed effect might e.g. only impact unintentional taps). But it is something to remain aware of.


 * Do they increase more or less for anons or editors, for editors per bucket?

Do page issues affect the time spent on each page?
...

What is the approximate percentage of (mobile) pageviews to pages with issues?
...

Other notes and caveats
...