Growth/Personalized first day/Variant testing

Starting in December 2019, the Growth team started testing variants of features against each other -- instead of just building one version of a feature and seeing whether it does or does not increase new editor retention. Testing variants will help us learn and iterate faster. This page contains running lists of variants we want to test for our different features, which we'll draw from when planning tests going forward. The lists are long and exhaustive, but we expect to test under 10 of these per year.

Newcomer tasks variants
Below is the list of feature variants we would be interested in testing for newcomer tasks. Competed tests have summarized results in the table and full results in the section below.

General homepage variants
Below is the list of feature variants we would be interested in testing for the newcomer homepage.

Initiation (A vs. B)

 * On desktop, Variant B yields double the interaction (clicking on anything in the suggested edits module), 60% more navigation (clicking on arrows to navigate to different suggested articles), and 30% more clicking on suggested articles.
 * On mobile, the variants perform the same for interaction and navigation, but Variant B leads to 15% less clicking on suggested articles.
 * On mobile, Variant A leads to 40% more newcomers editing than Variant B, but the edit rates are statistically the same on desktop.
 * Takeaways:
 * Making the suggested edits module more prominent on the homepage gets more users to interact with it.
 * Neither Variant A or B successfully made the module prominent on mobile.
 * Though the overlays were not present on Variant B, they may still play an important role in giving the user context for what they're supposed to be doing with the module. It's possible that this is reflected in the data in two ways:
 * Variant A yields 40% more newcomers editing on mobile than Variant B.
 * The increases between Variant A and B on desktop is lower with each successively "more engaged" type of engagement. In other words, simply interacting with the module is increased by 100%, but navigating between articles is only increased by 60%, and selecting an article is only increased by 30%.  This may show that users don't have sufficient context to understand that they are supposed to navigate between and select articles.
 * Next steps: we turned Variant A on for all newcomers while we build Variants C and D. This is because it is shown to yield more newcomers completing edits.
 * We'll use these learnings to design Variants C and D, with work happening here.