User:MWang (WMF)/Draft/Experiment analysis, December 2021

In November 2019, the Growth team added the "newcomer tasks" feature to the newcomer homepage. This feature was deployed in a controlled experiment where users were randomly assigned to either a control group that did not get access to any of the team's features, and a treatment group that did. We published an analysis of the effects of this feature in November 2020 based on data from the first six months after deployment.

We decided to gather new data and do another round of analysis focusing on the four key metrics from the first analysis: activation, retention, productivity, and revert rate. This was done for three reasons:


 * 1) The Guidance feature was not deployed during the original analysis.
 * 2) The original analysis used data from four wikis whereas in 2021 the Growth tools were on a lot more wikis.
 * 3) There were no significant changes to the Newcomer Tasks feature during this time period as the team was focusing on the Add a Link structured task.

We also took the opportunity to dig deeper into the effects of the Growth features by adding in data on whether a user appeared to be editing at the time of registration, and their responses to the Welcome Survey should they choose to respond to it. This provided us with new insight into areas where the features appear to be struggling as well as areas where they appear to be very successful.

Summary of Findings
In general, the analysis of key metrics find similar results as in the 2020 analysis except for productivity where we find no change. Specifically, the results are:


 * Newcomers who get the Growth features are more likely to be "activated" (i.e. making a first article edit within 24 hours of registration).
 * We strongly believe they are also more likely to be retained as editors (i.e. returning to the wiki to edit on a different day) as a result of being more active on their first day.
 * The features do not appear to increase or decrease productivity (i.e. number of article edits made), and the treatment and control groups make constructive edits at the same rate (i.e. the revert rate is the same).