Growth/Analytics updates/Help Panel experiment plan

The goal of the help panel is to provide users with easier ways to get help without them having to leave the editing context. This can then enable them to complete their tasks while potentially also meeting the Wikipedia community through questions and answers at the community’s help desk. Helping users complete their tasks can lead to an increase editor activation (the proportion of new users who make edits), and potentially also editor retention (the proportion of new users who return to edit a second time), the latter being the Growth Team's overarching goal.

In order to understand the help panel's impact on editor activation and retention, we propose a six month A/B test. During that test, 50% of new registrations on target wikis will have the help panel enabled by default, and 50% will have it disabled. We are likely to be running other experiments on the target wikis at the same time, for example testing variants of our welcome survey. If those experiments require stratified sampling, we will make sure our sampling strategies are modified accordingly.

For more information on the questions we intend to pursue with the help panel, see this section on the project page. For more information on exactly what data we will be recording, see this EventLogging schema.

Variants
During those six months we also envision testing variants of the help panel to understand how specific interface elements positively affect behaviors inside the workflow of seeking help. We will remember that the stronger our hypothesis that an altered interface will affect activation or retention, the more a test on that interface confounds our longer term experiment on activation and retention.

We will need to prioritize which of the variants to test first, because we will only test one at a time. We will also not run any of the variants until about a month of the larger experiment has run without any of these smaller ones nested inside. This is so that we can learn clearly whether the help panel itself has an effect on new user activation rate.

Leading indicators and plans of action
The duration of the A/B test is six months because it is impossible to detect changes to new editor retention on mid-size wikis in less time than that (unless we drastically impact retention, but we see that as somewhat unlikely). While we wait for our results we want to be able to take action if we suspect that something is amiss. Below, we sketch out a set of scenarios based on the data described in the instrumentation strategy above. For each scenario, we propose a plan of action to take to remedy the situation.

Status of leading indicators after one month
The Help Panel was deployed to Czech and Korean Wikipedia on January 11, 2019. One month later we gathered data for all registrations for both wikis up until that point. Known test accounts were removed, as were users who turned the Help Panel on or off in their preferences because self-selection into or out of the treatment group violates the equal expectation resulting from random assignment to groups. However, as we will see below, very few users changed their preferences.

There are four thresholds in the table above that are cause for concern. Of those, we have already followed up one of them, the help links in Czech Wikipedia. There, we suggested to remove the link to more information about notability and replaced it with a link about how to add an image, as the latter was the most frequent question posted to the Help Desk and the most frequently clicked link in Korean. We also suggested that the link to the guide be labelled "Quick tutorial", because that is what is used in the Korean Wikipedia, where that is the second most frequently clicked link. This work was tracked in T217391 and completed by March 6, 2019.

As for the three other thresholds, we are less concerned. The 25% open rate was likely too ambitious, and the same goes for the threshold of not asking questions. We are seeing relatively good usage numbers for the Help Panel, and therefore decided to not take any action until we also know the usage of its second version (that is shown also in reading mode and has search functionality). When it comes to the proportion of users who start the path to asking a question but do not complete it, we want to gather more data to get a better understanding. Here, we are limited by the usage numbers, and will therefore revisit this indicator at a later stage.