Reading/Web/Projects/Mobile Page Issues/AB tests

(...)

From September to November 2018, we ran A/B tests on the Latvian, Persian, Russian, Japanese, and English Wikipedias.

Does the new treatment increase the awareness among readers of page issues?
The clickthrough ratio (for top-of-page issues notices) increased markedly with the new treatment on all five wikis (e.g. over 7x on ruwiki). We can confidently assume that the new design increases the awareness of page issues among readers. ... (table and charts go here)

... ..
 * Is there an increase in clickthrough based on the new issue treatments (from the article page to the issues modal, from the issues modal to anywhere else - details about issues type, modal dismissed, etc, i.e. where do people go after the modal)?
 * Does clickthrough depend on the severity of each issue?

Do mobile edits increase with page issues as referrer?
For practical reasons, we limited ourselves to measuring taps on the edit button (rather than actual saved edits). (table and charts go here)

Based on the data we can reject the hypothesis that those edit button clicks would increase (on pages with issues) in the new design. To the contrary, we even saw a slight but statistically significant drop in edit button clickthroughs on four of the five wikis. Based on our current understanding, we don't yet regard this as evidence of a detrimental effect of the new design, e.g. considering thee absence of a clear explanation of a mechanism that could cause this (keeping in mind that what we could measure here are only taps on the button, not finished edits, so the observed effect might e.g. only impact unintentional taps). But it is something to remain aware of.


 * Do they increase more or less for anons or editors, for editors per bucket?

Do page issues affect the time spent on each page?
...

What is the approximate percentage of (mobile) pageviews to pages with issues?
...

Other notes and caveats
...