User:MWang (WMF)/Draft/Experiment analysis, December 2021

In November 2019, the Growth team added the "newcomer tasks" feature to the newcomer homepage. This feature was deployed in a controlled experiment where users were randomly assigned to either a control group that did not get access to any of the team's features, and a treatment group that did. We published an analysis of the effects of this feature in November 2020 based on data from the first six months after deployment.

We decided to gather new data and do another round of analysis focusing on the four key metrics from the first analysis: activation, retention, productivity, and revert rate. This was done for three reasons:


 * 1) The Guidance feature was not deployed during the original analysis.
 * 2) The original analysis used data from four wikis whereas in 2021 the Growth tools were on a lot more wikis.
 * 3) There were no significant changes to the Newcomer Tasks feature during this time period as the team was focusing on the Add a Link structured task.

We also took the opportunity to dig deeper into the effects of the Growth features by adding in data on whether a user appeared to be editing at the time of registration, and their responses to the Welcome Survey should they choose to respond to it. This provided us with new insight into areas where the features appear to be struggling as well as areas where they appear to be very successful.

Summary of Findings
In general, the analysis of key metrics find similar results as in the 2020 analysis except for productivity where we find no change. Specifically, the results are:


 * Newcomers who get the Growth features are more likely to be "activated" (i.e. making a first article edit within 24 hours of registration).
 * We strongly believe they are also more likely to be retained as editors (i.e. returning to the wiki to edit on a different day) as a result of being more active on their first day.
 * The features do not appear to increase or decrease productivity (i.e. number of article edits made), and the treatment and control groups make constructive edits at the same rate (i.e. the revert rate is the same).

We provide more details on the specific results below.

Glossary

 * As of December 2021, almost all Wikipedia wikis have the Growth features. During the data gathering for this analysis, the number of Wikipedia wikis with the features was still limited. We gathered data from 16 wikis from February 2021, 17 wikis from March 2021, and 28 wikis from April 2021.
 * Not all newcomers receive Growth features; 20% of them are randomly chosen to get the default experience. The group with the features is the treatment group and the group with the default experience is the control group. Numbers that come from the default experience are called baseline numbers.
 * Activation is defined as a newcomer making their first edit within 24 hours of registration. The baseline activation rate is the activation rate with the default features, not the Growth features.
 * Constructive activation is defined as a newcomer making their first edit within 24 hours of registration, and that edit not being reverted within 48 hours. The baseline constructive activation rate is the rate for users with the default features, not the Growth features.
 * Retention is defined as a newcomer coming back on a different day in the following two weeks after activation and making another edit. The baseline retention rate is the rate for users with the default features, not the Growth features. We can limit retention to constructive edits in a similar way as we did for activation, and then get a baseline constructive retention rate.
 * Edit volume is the overall count of edits made in a user's first two weeks. The baseline edit volume is the count for users with the default features, not the Growth features.

Detailed Findings
In this section we describe the specific impacts we've estimated from the controlled experiment. These are based on 244,060 new accounts registered across the wikis with the Growth features in February, March, and April 2021. For more specifics about our methodology, see "Methodology" below.





Activation
For this analysis, we focus on the Article and Article talk namespaces because 1) Newcomer Tasks is asking users to edit articles, and 2) the 2020 analysis found a significant positive effect on activation.


 * Activation: newcomers who get the Growth features are 2.3% more likely to make a first article edit. Across our dataset, the baseline activation rate in the Control group is 29.7%. In the Treatment group the activation rate is 30.4%, which is a 2.3% increase over the baseline.
 * Constructive activation: we find a larger effect of the Growth features when it comes to non-reverted edits in the Article and Article talk namespaces. Here the baseline constructive activation rate is 23.1%. The rate for users getting the Growth features is 23.8%, which is a 5.6% increase over the baseline.