Growth/Analytics updates/Help Panel experiment plan/ja

ヘルプパネルの目標はユーザーがページの編集作業を離れずに、もっと楽にヘルプを求める方法を提供することです. その効果としてタスクを完了できる上、コミュニティのヘルプペデスクに投稿して答えを得る過程でウィキペディアのコミュニティとの出会いのチャンスが訪れます. ユーザーのタスク完了に手を貸すと、活発な編集者 (新人ユーザーに占める編集作業をする人の割合) の増加に結びつき、さらには編集者の定着 (新人ユーザーで2回目以降の編集作業をしにウィキペディアを再訪した人の割合) を促進する可能性があり、後者は Growth チームの包括的なゴールでもあります.

ヘルプパネルが編集者の活動ならびに定着率に与える影響を理解するには、6カ月をかけた A/B 試験の実施を提案します. 試験中に対象ウィキ群の新規登録者の 50% は既定でヘルプパネルが利用できるようにして、残り 50% には利用できなくします. 対象のウィキ群では新人歓迎アンケートの複数案比較など、同時に複数の実験を行う場合を想定しています. それらの実験で層化抽出法を採用する場合は、標本の抽出法の調整に留意します.

このチームがヘルプパネルを使って明らかにしたい質問の詳細は、 プロジェクトページのこちらの節を参照してください. 実際に記録するデータの詳細はこちらの EventLogging スキーマで説明しています.

Variants
試験期間の6カ月には、ヘルプを求めるワークフローにおいて個別のインターフェース要素が行動にどのようなよい影響を与えるか理解するため、ヘルプパネルのいくつかのバリエーションを試験したいと考えています. インターフェース次第で (編集者の) 活発さや定着率が左右されるという仮説にこだわりすぎると特定のインターフェースばかり試験してしまい、長い目で見た活発さと定着率の試験を誤った結果に導きかねない点には気をつけます.

We will need to prioritize which of the variants to test first, because we will only test one at a time. We will also not run any of the variants until about a month of the larger experiment has run without any of these smaller ones nested inside. This is so that we can learn clearly whether the help panel itself has an effect on new user activation rate.

Leading indicators and plans of action
The duration of the A/B test is six months because it is impossible to detect changes to new editor retention on mid-size wikis in less time than that (unless we drastically impact retention, but we see that as somewhat unlikely). While we wait for our results we want to be able to take action if we suspect that something is amiss. Below, we sketch out a set of scenarios based on the data described in the instrumentation strategy above. For each scenario, we propose a plan of action to take to remedy the situation.

導入1カ月後の先行指標の状態
The Help Panel was deployed to Czech and Korean Wikipedias on January 11, 2019. One month later we gathered data for all registrations for both wikis up until that point so that we can evaluate our leading indicators and determine whether any of the feature's behavior was concerning. In short, while the evaluation exposes some areas for improvement, we think the help panel's behavior so far is healthy and that it is not having a negative impact on the wikis.

Known test accounts were removed, as were users who turned the Help Panel on or off in their preferences because self-selection into or out of the treatment group violates the equal expectation resulting from random assignment to groups. However, as we will see below, very few users changed their preferences.

There are four thresholds in the table above that are cause for concern, and the list below explains how we're thinking about them.


 * Not opening the help panel: for both wikis, the number is somewhat, but not alarmingly higher than the threshold. This makes us feel like we have healthy open rates that have room for improvement.  In that vein, we have started to display the help panel in more places, so that users have more opportunities to open it up.  The help panel is now being displayed in reading mode in the Help, Wikipedia, and User namespaces.  This work was tracked in T215664 and completed by March 6, 2019.


 * Not clicking help links: since the analysis showed that three Czech links were getting low traffic, we removed the link to more information about notability and replaced it with a link about how to add an image, as the latter was the most frequent question posted to the Help Desk and the most frequently clicked link in Korean. We also relabelled the link to the guide to be labelled "Quick tutorial", because that is what is used in the Korean Wikipedia, where that is the second most frequently clicked link. This work was tracked in T217391 and completed by March 6, 2019.


 * Not asking questions: in retrospect, our expectation that 75% of users who open the help panel (and don't click links) would ask a question was likely too ambitious. We feel comfortable with the rate of questions being asked in Czech, and we are learning that perhaps the paradigm of asking public questions is not a great fit for Korean Wikipedia, given the low rate of questions there.  This is one of the reasons we added "search" to the help panel, so that users would have different ways to find help that might fit their own preferences of how to find it.  This work was tracked in T209301 and completed by February 25, 2019.


 * Starting the question path and not completing it: the absolute numbers for this metric are still low enough that it is hard to say with confidence whether we are notably higher than the threshold. At the time of this analysis, only about 25 people had attempted asking a question.  We will therefore revisit this indicator at a later stage.