Growth/Analytics updates/Welcome survey editor activation rate

From MediaWiki.org
Jump to navigation Jump to search

Summary[edit]

When we deployed the "welcome survey" to Czech and Korean Wikipedia, our main concern was that it would lead to fewer newly registered users becoming editors within the first 24 hours after registering (what we call "editor activation"). This concern is described in our experiment plan, and is why we ran the survey as a randomized A/B test over the course of a month: half of new registrations received the survey immediately after creating their account, half did not get the survey at all and returned to the context they were in before creating their account. In this update, we describe the results of that A/B test, answering the question: does having the welcome survey affect editor activation rate? We find that there does not appear to be a statistically significant difference in activation rate between the survey and control groups in either of the two Wikipedias.

In testing Arabic Wikipedia in August 2019, we also find that there is no statistically significant difference in activation rate between the survey and control groups.

Next, the Growth team will test a more heavily designed version of the survey, Variation C, against the simple Variation A tested in the experiment discussed on this page.

Background[edit]

The survey was deployed to Czech and Korean Wikipedias on November 19, 2018, shortly after 19:00 UTC. In this analysis, we use data from deployment until December 25, so as to use whole weeks.[footnote 1] While we had one more week of data available at the time of analysis, we discarded it due to a spambot attack affecting registrations on Korean Wikipedia.

In addition to limiting accounts by date of registration, we also apply several other filters:

  • First, the survey is only shown to users who register on the given wiki, so we filter out users who already had accounts on another wiki (also known as "autocreated accounts").
  • Secondly, we filter out accounts created through Wikipedia's API as those are mainly accounts created from the Wikipedia Android and iOS apps, and the survey is not running on either of those apps.
  • Lastly, we remove known test accounts created by Growth Team members.

Results[edit]

Our dataset contains 1,617 accounts in Czech Wikipedia and 2,140 accounts in Korean Wikipedia. For each of these accounts, we use Wikipedia's edit history to calculate whether a user made at least one edit within 24 hours after registration. We only calculate this for the first 24 hours because our previous analysis revealed that users who become editors are most likely making that transition quickly, only about 10% of those who ever make an edit make their first edit later than 24 hours after registration. With data on whether they edit, and whether they were shown the welcome survey or not, we can create 2x2 contingency matrices for both wikis. They are shown in Tables 1–4 below.

Table 1: Activation counts by group, Czech
Experiment group Did not edit Made ≥ 1 edit Total
Control 459 342 801
Survey 441 375 816
Total 900 717 1,617
Table 2: Activation % by group, Czech
Experiment group Did not edit Made ≥ 1 edit Total
Control 57.3% 42.7% 100.0%
Survey 54.0% 46.0% 100.0%
Total 55.7% 44.3% 100.0%
Table 3: Activation counts by group, Korean
Experiment group Did not edit Made ≥ 1 edit Total
Control 618 451 1,069
Survey 658 413 1,071
Total 1,276 864 2,140
Table 4: Activation % by group, Korean
Experiment group Did not edit Made ≥ 1 edit Total
Control 57.8% 42.2% 100.0%
Survey 61.4% 38.6% 100.0%
Total 59.6% 40.4% 100.0%

Note: in Tables 2 & 4, proportions are calculated per row. This is to make it easier to compare activation rates for the survey and control groups.

The proportions shown in Tables 2 & 4 suggests conflicting trends between the survey and control groups in the two wikis. In Czech, the survey group has a slightly higher activation proportion (by 3.3pp), while in Korean it is slightly lower (by 3.6pp). But, are any of these differences statistically significant?

The answer is "no". Using a two-sample test of equality in proportions we find that neither the difference in Czech () nor Korean () is statistically significant.

Addendum: Arabic Wikipedia[edit]

We deployed the Welcome Survey to Arabic Wikipedia on July 15, 2019, using the same A/B test configuration we used for Czech and Korean. Users who signed up on Arabic Wikipedia would be randomly assigned to either a survey or control group with 50% probability. In our analysis of the results we used five whole weeks of data, meaning all registrations up until August 26, 2019. As we did for Czech and Korean, auto-created accounts are removed from our dataset because the survey is only shown to local registrations. We also remove accounts created through the API (these are primarily app accounts), and known test accounts.

Our dataset contains 9,524 accounts, of which 4,808 (50.5%) were in the control group, and 4,716 (49.5%) were in the survey group. Similarly as for Czech and Korean Wikipedia, we use the MediaWiki databases to count how many edits these users made in the first 24 hours after registration. From this data we can then create 2x2 contingency tables, seen in Tables 5 & 6 below.

Table 5: Activation counts by group, Arabic
Experiment group Did not edit Made ≥ 1 edit Total
Control 3,469 1,339 4,808
Survey 3,372 1,344 4,716
Total 6,841 2,683 9,525
Table 6: Activation % by group, Arabic
Experiment group Did not edit Made ≥ 1 edit Total
Control 72.2% 27.8% 100.0%
Survey 71.5% 28.5% 100.0%
Total 71.8% 28.2% 100.0%

Note: in Table 5 proportions are calculated per row. This is to make it easier to compare activation rates for the survey and control groups.

There is a small difference in activation between the control and the survey group, but this difference is not statistically significant (). Therefore, we conclude that the welcome survey does not negatively impact whether newcomers stay on the wiki and make contributions.

Footnotes[edit]

  1. Wikipedia activity tends to fluctuate in weekly cycles, see for example Yasseri, Taha, Robert Sumi, and János Kertész. "Circadian patterns of wikipedia editorial activity: A demographic analysis." PloS one 7.1 (2012): e30091.