Growth/Analytics updates/Welcome survey editor activation rate

When we deployed the Welcome survey to Czech and Korean Wikipedia, our main concern was that it would lead to fewer newly registered users become editors within the first 24 hours after registering (what we call "editor activation"). This concern is described in our experiment plan, and is why we ran the survey as a randomized A/B test over the course of a month. Half of new registrations got the survey, half did not. In this update, we describe the results of that A/B test, answering the question: does the welcome survey affect editor activation rate? We find that there does not appear to be a statistically significant difference in activation rate between the survey and control groups in either of the two Wikipedias.

The survey was deployed to Czech and Korean Wikipedias on November 19, 2018, shortly after 19:00 UTC. In this analysis, we use data from deployment until December 25, so as to use whole weeks. While we had one more week of data available at the time of analysis, we discarded it due to a spambot attack affecting registrations on Korean Wikipedia.

In addition to limiting accounts by time of registration, we also apply several other filters. First, the survey is only shown to users who register on the given wiki, so we filter out users who already had accounts on another wiki (also known as "autocreated accounts"). Secondly, we filter out accounts created through Wikipedia's API as those are mainly app accounts, and the survey is not running on either of the Wikipedia apps. Lastly, we remove known test accounts created by Growth Team members.

Our dataset contains 1,617 accounts in Czech Wikipedia, and 2,140 accounts in Korean Wikipedia. For each of these accounts, we use Wikipedia's edit history to calculate whether a user made at least one edit within 24 hours after registration. We only calculate this for the first 24 hours because our previous analysis revealed that users who become editors are most likely making that transition quickly, only about 10% of them make their first edit later than that. With data on whether they edit, and whether they were shown the welcome survey or not, we can create 2x2 contingency matrices for both wikis. They are shown in Tables 1–4 below. Note: in Tables 2 & 4, proportions are calculated per row. This is to make it easier to compare activation rates for the survey and control groups.

The proportions shown in Tables 2 & 4 suggests conflicting trends between the survey and control groups in the two wikis. In Czech, the survey group has a slightly higher activation proportion (by 3.3pp), while in Korean it is slightly lower (by 3.6pp). But, are any of these differences statistically significant?

The answer is "no". Using a two-sample test of equality in proportions we find that neither the difference in Czech ($$\Chi^2=1.61, df=1, p=0.20$$) nor Korean ($$\Chi^2=2.77, df=1, p=0.095$$) is statistically significant.