Growth/Analytics updates/Personalized first day experiment plan

Our Personalized first day intervention will deploy a short survey to users who register on our target wikis. In preparation for that going live, we are considering whether to run any experiments with that survey, and if we do, what type of experiments to run. We started out by discussing this in T206380, but then turned it into a larger plan that doesn't fit Phabricator well. This page describes and explains our current plans. Feel free to follow up on the |talk page with any questions or comments you might have, this is a work in progress and we welcome feedback!

Experiment plan
The goal of the intervention is to learn more about those who create accounts on our target wikis, and we plan to use a short survey to do so. That changes the signup process by introducing another step and asking the user to spend some time answering the questions in the survey. This could change user behavior, for instance by distracting the user from what they were doing prior to creating an account, or changing their motivation to continue spending time on Wikipedia.

The Growth Team's overarching goal is to increase editor retention, and the first part of becoming a retained editor is to become an editor. Our primary concern regarding the survey is therefore whether it leads to fewer new users making their first edit. This has led us to propose a two-step experiment plan for this survey:


 * 1) A one-month A/B test with the survey in target wikis to gather data and measure its impact on editor activation.
 * 2) If the survey does not significantly affect editor activation rates, deploy it to a larger proportion of new users in order to gather more responses.

In the A/B test, we show the survey to 50% of newly registrations. The remaining 50% become our control group. Even though we are concerned that the survey might reduce editor activation, we choose a 50/50 split because it allows us to determine if there is an effect as quickly as possible. Secondly, we will discuss several other measurements below that might provide us with indications that our survey is not functioning the way we intended.