Growth/Personalized first day/Structured tasks/Newcomer task edit type analysis, April 2022

From mediawiki.org

The Growth team deployed Add a Link, the first structured task in late May 2021. The second task, Add an Image, followed in late November 2021. A concern about structured tasks has been that they result in newcomers only doing those types of edits, meaning they do not progress towards other types of edits.

In this analysis we use data from wikis where Add a Link was deployed and examine whether newcomers who start out with Add a Link edits end up continuing to only make those types of article edits. We also do the same for users who started out with a different Newcomer Tasks edit so we can compare between the two groups. We find that both groups are more likely to go on to make additional article edits compared to those who do not start out with a Newcomer Tasks, and that more than 70% of newcomers who start out with Add a Link go on to make a combination of Newcomer Tasks and other article edits or only other kinds of article edits.

We dig into this in more detail below and the Methodology section describes how we gathered data for this analysis.

Making additional article edits[edit]

Treemap illustrating the likelihood of a second edit session per Table 1, with a breakdown of the types of article edits made in correspondence with Table 2.
Figure 1: Likelihood of a second edit session for newcomers who started with Add a Link or Other Newcomer Tasks.

We first examine to what extent users who start out with Newcomer Tasks go on to make additional article edits. The statistics are shown in Table 1 and illustrated in Figure 1. We can see from the table that newcomers who start out with a Newcomer Tasks edit (first row, 25.4%) are more likely to go on to make additional article edits than those who do not (third row, 23.5%). We can also see that newcomers who start out with Add a Link are somewhat less likely to make additional article edits. In this case we have not determined whether any of these differences are statistically significant because this is not the focus of this analysis. When it comes to Add a Link's effect on newcomers' editing we instead refer to our experiment analysis from December 2021 where we found strong positive effects.

Table 1: Likelihood of Additional Article Edits
First article edit Additional article edits
Newcomer Tasks 1,905 483 25.4%
Add a Link 984 246 25.0%
Other 17,534 4,118 23.5%
Total 19,439 4,601 23.7%

Types of additional article edits made[edit]

As mentioned in the introduction the main focus of this analysis is to understand what types of article edits newcomers go on to make. The main concern being that users who start out with Add a Link will be "locked in" to that structured task and not learn how to make other types of article edits. Table 2 shows the statistics for what types of article edits newcomers go on to make, and this is also illustrated in the top three squares for each group in Figure 2. We can see that in both cases, the majority of newcomers go on to make either a combination of both Newcomer Tasks and other edits (third row) or do not make any additional Newcomer Tasks edits (first row).

Table 2: Type of Additional Article Edits
Type of article edits Add a Link Other Newcomer Tasks
No Newcomer Tasks 85 34.6% 106 44.7%
Only Newcomer Tasks 69 28.0% 44 18.6%
Both 92 37.4% 87 36.7%
Total 246 100.0% 237 100.0%

While the proportion of newcomers who make only Newcomer Tasks article edits after starting with an Add a Link edit (28.0%) is higher than for those who start out with other Newcomer Tasks (18.6%), we did investigate this group further and found that less than half of them (44.9%) only made Add a Link edits. Based on these results, we conclude that we do not find evidence suggesting that newcomers who start out with Add a Link continue to only make those kinds of edits.

Methodology[edit]

We use newcomers registered on any of the 11 wikis where Add a Link was deployed (Arabic, Bengali, Czech, Vietnamese, Persian, French, Hungarian, Polish, Romanian, Russian, and Spanish) during the month of April 2022. This month was chosen because it was the first whole month where all types of newcomer task edits had distinct edit tags, which enables us to investigate the type of Newcomer Tasks edit made in more detail. Given the time window of 37 days that we will describe shortly, it was also the only month for which we would have full data for all new registrations.

We restricted eligible accounts using the typical set of limitations for Growth team analyses: we require users to have self-registered on the given wiki and known test accounts and bot accounts are excluded. We chose not to exclude API registrations this time because MediaWiki history does not have that information, and to keep the analysis time short we did not correlate registrations with other dataset.

We define a “newcomer” as a registered user who makes an article edit within one week of registration, and identify whether that first edit was a newcomer task edit. If it was such an edit we also identify whether it was an Add a Link edit. This means that we do not separate users into treatment and control categories, but instead let users self-select based on what their first edit is. We do this because we are not particularly concerned with learning a “true” baseline of edit probabilities, we are instead focusing on what users are doing during their early days on the wiki.

We examine these article editing newcomers’ possible additional edits and require them to have a second edit session within 30 days of the first, meaning that there has to be at least one hour between two consecutive edits. Many newcomers make multiple consecutive edits at the start, e.g. because they noticed they made a typo. By combining those into a single session we do not let the other edits count, meaning we instead learn about what they did the next time they came to the wiki.

For all additional article edits starting with the first one of the second edit session, we count how many they did, whether they were newcomer task edits, and if so what specific newcomer task they were. This allows us to categorize newcomers by what kind of article edit they started out with as well as what kind of article edits they continued to do over the next 30 days.