Growth/Analytics updates/Welcome survey editor activation rate

From MediaWiki.org
Jump to navigation Jump to search

Summary[edit]

When we deployed the "welcome survey" to Czech and Korean Wikipedia, our main concern was that it would lead to fewer newly registered users becoming editors within the first 24 hours after registering (what we call "editor activation"). This concern is described in our experiment plan, and is why we ran the survey as a randomized A/B test over the course of a month: half of new registrations received the survey immediately after creating their account, half did not get the survey at all and returned to the context they were in before creating their account. In this update, we describe the results of that A/B test, answering the question: does having the welcome survey affect editor activation rate? We find that there does not appear to be a statistically significant difference in activation rate between the survey and control groups in either of the two Wikipedias.

Next, the Growth team will test a more heavily designed version of the survey, Variation C, against the simple Variation A tested in the experiment discussed on this page.

Background[edit]

The survey was deployed to Czech and Korean Wikipedias on November 19, 2018, shortly after 19:00 UTC. In this analysis, we use data from deployment until December 25, so as to use whole weeks.[footnote 1] While we had one more week of data available at the time of analysis, we discarded it due to a spambot attack affecting registrations on Korean Wikipedia.

In addition to limiting accounts by date of registration, we also apply several other filters:

  • First, the survey is only shown to users who register on the given wiki, so we filter out users who already had accounts on another wiki (also known as "autocreated accounts").
  • Secondly, we filter out accounts created through Wikipedia's API as those are mainly accounts created from the Wikipedia Android and iOS apps, and the survey is not running on either of those apps.
  • Lastly, we remove known test accounts created by Growth Team members.

Results[edit]

Our dataset contains 1,617 accounts in Czech Wikipedia and 2,140 accounts in Korean Wikipedia. For each of these accounts, we use Wikipedia's edit history to calculate whether a user made at least one edit within 24 hours after registration. We only calculate this for the first 24 hours because our previous analysis revealed that users who become editors are most likely making that transition quickly, only about 10% of those who ever make an edit make their first edit later than 24 hours after registration. With data on whether they edit, and whether they were shown the welcome survey or not, we can create 2x2 contingency matrices for both wikis. They are shown in Tables 1–4 below.

Table 1: Activation counts by group, Czech
Experiment group Did not edit Made ≥ 1 edit Total
Control 459 342 801
Survey 441 375 816
Total 900 717 1,617
Table 2: Activation % by group, Czech
Experiment group Did not edit Made ≥ 1 edit Total
Control 57.3% 42.7% 100.0%
Survey 54.0% 46.0% 100.0%
Total 55.7% 44.3% 100.0%
Table 3: Activation counts by group, Korean
Experiment group Did not edit Made ≥ 1 edit Total
Control 618 451 1,069
Survey 658 413 1,071
Total 1,276 864 2,140
Table 4: Activation % by group, Korean
Experiment group Did not edit Made ≥ 1 edit Total
Control 57.8% 42.2% 100.0%
Survey 61.4% 38.6% 100.0%
Total 59.6% 40.4% 100.0%

Note: in Tables 2 & 4, proportions are calculated per row. This is to make it easier to compare activation rates for the survey and control groups.

The proportions shown in Tables 2 & 4 suggests conflicting trends between the survey and control groups in the two wikis. In Czech, the survey group has a slightly higher activation proportion (by 3.3pp), while in Korean it is slightly lower (by 3.6pp). But, are any of these differences statistically significant?

The answer is "no". Using a two-sample test of equality in proportions we find that neither the difference in Czech () nor Korean () is statistically significant.

Footnotes[edit]

  1. Wikipedia activity tends to fluctuate in weekly cycles, see for example Yasseri, Taha, Robert Sumi, and János Kertész. "Circadian patterns of wikipedia editorial activity: A demographic analysis." PloS one 7.1 (2012): e30091.