Growth/Personalized first day/Structured tasks/Add a link/Experiment analysis, December 2021

In May 2021, the Growth team added the "Add a Link" structured task to the "newcomer tasks" feature on the newcomer homepage. Structured tasks are ones that can be broken down into step-by-step workflows with simple steps that make sense to newcomers, and that are easy to do on mobile devices. Our hypotheses were that users would be more likely to complete Add a Link tasks than the unstructured task, and that this would result in an increase to our core metrics without decreasing edit quality.

To learn more about the impact of Add a Link, we deployed the feature in a controlled experiment. Users were randomly assigned to one of three groups: a control group that did not get access to any of the team's features (20%), a group that got access to the Growth features and where the default task assigned in the "newcomer tasks" module was Add a Link (40%), and a group that got access to the Growth features with the default task assigned being the unstructured "add links" task (40%). The experiment started on May 27, 2021 with four Wikipedias: Arabic, Bengali, Czech, and Vietnamese. Six additional Wikipedias were added on July 21 of the same year: Russian, French, Polish, Romanian, Persian, and Hungarian. Our analysis uses data from these wikis to October 14, 2021.

Summary of findings
In general, the analysis showed that the Add a Link structured task improves outcomes for newcomers. The most important points are:


 * Newcomers who get the Add a Link structured task are more likely to be "activated" (i.e. make a first article edit).
 * They are also more likely to be retained (i.e. come back and make another article edit on a different day).
 * The feature also increases edit volume (i.e. the number of edits made across the first couple weeks), while at the same time improving edit quality (i.e. the likelihood that the newcomer's edits are reverted).

Glossary

 * As of mid-March 2022, eleven wikis have the Add a Link structured task. In our experiment we analyze data from ten of them: Arabic, Bengali, Czech, Vietnamese, Russian, French, Polish, Romanian, Persian, and Hungarian.
 * Not all newcomers received the Add a Link structured task: 20% of users were randomly chosen to get the default newcomer experience, which does not have access to any of the Growth features. We refer to this group as the control group. 40% were randomly chosen to get the Growth features with the default task in the newcomer tasks module set to the Add a Link structured task, and we call that group the Add a Link group. The remaining 40% were also given the Growth features but with no access to the Add a Link structured task and the default task in the newcomer tasks module set to the unstructured link task. We refer to this group as the unstructured task group.
 * Constructive activation is defined as a newcomer making their first edit within 24 hours of registration, and that edit not being reverted within 48 hours. The baseline constructive activation rate is the rate of constructive activation for the control group.
 * Activation is similarly defined as constructive activation, but without the non-revert requirement.
 * Constructive retention is defined as a newcomer coming back on a different day in the two weeks after constructive activation and making another edit, with said edit also not being reverted within 48 hours.
 * Retention is similarly defined as constructive retention, but without the non-revert requirements.
 * Constructive edit volume is the overall count of edits made in a user's first two weeks, with edits that were reverted within 48 hours removed. The baseline constructive edit volume is the count for users in the control group.
 * Revert rate is the proportion of edits that were reverted within 48 hours out of all edits made. This is by definition 0% for users who made no edits, and we generally exclude these users from the analysis.

Detailed findings
Below are the specific impacts estimated from the controlled experiment. These are based on observing 130,179 new accounts on the ten wikis between May and October 2021. For more specifics on the methodology, see "Methodology" below.





Activation
In this analysis, we focus on the Article and Article talk namespaces because 1) the Add a Link task asks users to edit articles, and 2) we've seen significant positive effects on activation in related analyses.


 * Activation: newcomers who get the Add a Link structured task are 11.7% more likely to make a first article edit compared to the baseline. Across our dataset, the baseline activation rate in the Control group is 27.2%. The activation rate for users who get the Add a Link task is 30.4%, which is an 11.7% relative increase over the baseline. Users in the unstructured task group have an activation rate of 28.6% (5.4% relative increase over the baseline), a difference that is both significantly lower than the Add a Link group and significantly higher than the control group.
 * Constructive activation: we find a larger effect of the Growth features when it comes to non-reverted activation edits. For the Control group the baseline constructive activation rate is 20.7%. The rate in the Add a Link group is 24.2%, which is a 16.6% relative increase over the baseline. Users who get the unstructured link task have a constructive activation rate of 22.0% (5.9% relative increase over the baseline), which is again a difference that is both significantly higher than the Control group and significantly lower than the Add a Link group.

When measuring across all namespaces, we find that the Control group and Unstructured task group have almost identical activation rates. The baseline activation rate in the Control group is 37.7%, and the baseline constructive activation rate is 31.4%. The activation rate in the Add a Link group is significantly higher in both cases: overall activation rate 39.0%, which is a relative increase of 3.6%; constructive activation rate 33.1%, which is a relative increase of 5.4%.