Growth/Analytics updates/Key metrics and power analysis
The Growth Team is planning on running interventions aiming to increase new editor retention, as stated in our goals and metrics. However, the standard way of measuring retention (Surviving new editor) uses a two month window. We would prefer to be able to understand the impact of our interventions sooner than two months. This led us to dig more closely into our data on new contributors to our target Wikipedias (Czech and Korean) to understand what the typical contributor behavior is. Next, we used those insights to define three key metrics, followed by a power analysis to estimate the duration of our interventions depending on their magnitude of change.
Our key metrics relate strongly to the definition of "surviving new editor" linked above, but define the path towards retention more clearly and with specific time constraints. Where "surviving new editor" typically has an loosely specified period of time for exploration, our metrics define a clear three-step path: activation, short-term retention, and long-term retention. These three steps form a, each being a requirement for the next. This means that a long-term retained editor also is a short-term retained editor, and similarly that any retained editor was once activated.
Next, we'll walk through our three key metrics and describe more about how we came to define them.
Activation Rate[edit]
In order to be a retained editor, a user first needs to become an editor. We call this event "activation". One way to increase the number of retained editors might be to increase the number of users who become editors, thus one of our key metrics is the proportion of registered accounts who make an edit.
The question is, how much time should we give them to make that edit? To answer that question, we analyzed data from our two target Wikipedias, Czech and Korean. We looked at both historical data (going back to the beginning of 2006) as well as recent data (January 2016 onwards) in order to understand if there has been any significant changes recently. We did not find differences in the overall trend, which we will describe in more detail below, instead there are small changes in the proportions.
Looking at the time it takes for a newly registered user to become an editor ("time to first edit") for non-autocreated accounts that make an edit within the first year, we can break it down as seen in the two tables below. This data is from Jan 1, 2016 through June, 2017 and is from the Czech Wikipedia. For simplicity, months have 30 days except for the last which has 35 to make a year have 365 days. The proportions are slightly different in Korean, something we'll discuss more below, but the overall trends are similar.
| First Year 10,018 100% | |||||
| 1–6 months 9,886 98.7% |
7–12 months 132 1.3% | ||||
| 1–3 months 9,734 97.2% |
4–6 months 152 1.5% |
7–9 months 82 0.8% |
10–12 months 50 0.5% | ||
| Month 1 9,460 94.4% |
Month 2 173 1.7% |
Month 3 101 1.0% |
|||
| Week 1 9,084 90.7% |
Week 2 164 1.6% |
Week 3 103 1.0% |
Week 4 89 0.9% | ||||||
| Day 1 8,581 85.7% |
Day 2 143 1.4% |
Day 3 94 0.9% |
Day 4 58 0.6% |
Day 5 67 0.7% |
Day 6 62 0.6% |
Day 7 79 0.8% |
|||