Growth/Personalized first day/Newcomer tasks/Experiment analysis, November 2020

In November 2019, the Growth team added the "newcomer tasks" feature to the newcomer homepage for the wikis we work with. Newcomer tasks provide a feed of suggested articles to edit, tailored to the area of interests of the newcomer. The objective was to give newcomers easy edits that they are interested in when they first arrive on the wiki. We believed that this would make it more likely for newcomers to start editing, to learn editing skills, see their impact, and then continue editing.

Six months after the feature was added to the newcomer homepage, we then collected data from Arabic, Vietnamese, Czech, and Korean Wikipedias to analyze the overall impact of the Growth features, including newcomer tasks. This was possible because the Growth features are deployed in a controlled experiment, in which 80% of newcomers have the features and 20% do not.

Summary of findings
In general, the analysis showed that the Growth features improve outcomes for newcomers. Below are the most important points.

We believe that these results validate that the Growth features, especially newcomer tasks, cause newcomers to edit more and stay on the wiki for longer. Therefore, we think all Wikipedias should consider implementing these features. We also believe that these results validate that the Growth team should continue our work on structured tasks, meant to create new kinds of easy editing workflows for newcomers.
 * Newcomers who get the Growth features are more likely to be activated (i.e. make a first article edit).
 * We believe they are also more likely to be retained (i.e. come back and make another article edit on a different day).
 * The features also increase edit volume (i.e. number of edits) and constructiveness (i.e. whether edits are reverted).

Detailed findings
Below are the specific impacts we estimated from the controlled experiment. These are all based on observing 97,755 new accounts in Arabic, Vietnamese, Czech, and Korean Wikipedias between November 2019 and May 2020. For more specifics on the experiment design, see "experiment design" below.

Activation
Activation is defined as a newcomer making their first edit. For this experiment, we focused on edits to the Article and Article Talk namespaces.
 * Activation: newcomers with Growth features are 11.6% more likely to make a first article edit.  The baseline activation rate across the four wikis in the experiment is 21.6%. The Growth features bring this to 24.1%.


 * Constructive activation : the effect is larger when looking only at constructive activation. Constructive activation is a newcomer making their first unreverted edit.  Newcomers with Growth features are 26.7% more likely to make a first constructive article edit.  The baseline constructive activation rate across the four wikis in the experiment is 16.1%.  The Growth features bring this to 20.4%.  This means that the features nudge newcomers toward edits that are less likely to be reverted.



Retention
Retention is defined as a newcomer who has edited then coming back on a different day in the following two weeks and making another edit. Because retention is much more rare than activation, it is harder to detect changes. Though we do not detect it directly, we estimate that retention is increased to a similar degree that activation is increased, i.e. by about 11.6%. The idea is that the Growth features primarily impact activation, in that they get more people to make their first edit. But then those additional people who made their first edit are retained at similar rates to others, leading to an increase in retention caused by the increase in activation. The baseline retention rate across the four wikis in the experiment is 3.2%. We estimate that the Growth features bring this to 3.6%.



Edit volume
The Growth features lead to an 85.3% increase in the number of article edits by newcomers in their first two weeks. The baseline average number of article edits made by newcomers in the experiment's four wikis is 0.34. Newcomers with the Growth features make an average of 0.63 article edits. In other words, 1,000 newcomers without the Growth features would make 340 article edits, while 1,000 newcomers with the Growth features would make 630. This large increase is associated with a small number of newcomers who make many suggested edits quickly, sometimes over 100 within two weeks of registration.

Other metrics
In addition to the main metrics above, we looked at several other ones, with less interesting findings.


 * Reverts: we looked at whether newcomers with Growth features were more or less likely to have their edits reverted.  This analysis did not show large or clear results.
 * Highly active newcomers: while our results have shown that Growth features cause more newcomers to become active and to make more edits, we also wanted to see whether the features lead to more newcomers becoming highly active, which we defined as making 50 edits in their first 30 days.  This analysis did not show differences resulting from the Growth features.
 * Thanks: we looked at whether newcomers with Growth features receive more “thanks” than other newcomers.  We found similar results to the retention analysis, in that we expect that Growth features do lead to more thanks received, but only because they cause more edits -- not because they cause newcomers to make edits that are more likely to attract thanks.
 * Differences between wikis and platforms: no analyses found significant differences in the effect of the Growth features between wikis or platforms.

Takeaways

 * Features work: the Growth team features work to increase newcomer engagement, especially the "newcomer tasks" component, which suggests easy edits.
 * Confidence in building structured tasks: this gives us confidence that our current work to build more kinds of newcomer tasks, such as the "add a link" task, will increase impact.
 * Need for positive reinforcement: the results showed that the Growth features primarily impact activation -- getting newcomers to make their first edit -- as opposed to retention. The features only seemed to increase retention because they increased activation.  Therefore, the Growth team should think about what we can add to the features to encourage newcomers to return after making their first edits.  Therefore, we are planning work on "positive reinforcement" this year, which will add milestones and statistics so that newcomers can get excited about their progress and impact.

Next steps

 * Spread the features: now that we have increased confidence in the value of the features, the Growth team will encourage more wikis to look over these results and consider deploying the features.
 * Continuing the work: this year, we'll continue to focus on adding new types of tasks and providing positive reinforcement when newcomers complete tasks.
 * Extend the analysis: having already completed this analysis, we're able to more easily run it again in the future. We'll be able to look at how the features impact more wikis, and see how improvements alter their impact.

Experiment designs
The Growth Team deployed the newcomer tasks module to the Homepage on Czech, Korean, Vietnamese, and Arabic Wikipedias on November 21, 2019. During the experiment, users were randomly assigned to either a treatment or control group. In the treatment group, users received all Growth features (homepage, newcomer tasks, help panel, etc.), while users in the control group received none.

From November 21 until December 12, 2019, the chance of being in the treatment group was 50%. This changed to 80% on December 12, when the team started an A/B test of two variants of the newcomer tasks module.

Users can turn the Growth features on or off in their user preferences at any point. If doing so, they are excluded from this analysis. We also exclude known test accounts, users who registered through the API (these are mainly app accounts), and accounts that are autocreated.

The dataset for this analysis contains 97,755 accounts registered between the start of the experiment and May 14, 2020. Of these, 23,529 (24.1%) are in the control group and 74,226 (75.9%) are in the treatment group.

[PARAGRAPH ABOUT HIERARCHICAL MODELING]