Growth/Focus on help desk/ja

このページではGrowth チームが担当する「ヘルプデスクに注目」プロジェクトを説明し、主なアセットや設計、意思決定を載せてあります. 進捗状況に関する定期的な更新情報は主にGrowth チームの更新ページにあり、このページには大がかりな、あるいは詳細な更新情報があります.

"ヘルプパネル"とはこのプロジェクトを支えるために開発された機能で、詳細はこのページで説明してあります. そのインタラクティブな試用版link to an interactive prototypeをここにご紹介します. 誰でもテスト ウィキペディアでヘルプパネルを実際に試用でき、アカウントの登録後、 個人設定/編集を開いてヘルプパネルを有効にしてからどこかのページで「編集」ボタンを押してください.

2019年9月時点で試行結果の分析を完了、ヘルプパネルは新人編集者の活発化にも定着率向上にも影響しなかった ことが示されました. それにより、この機能は今後、反復作業を進めないことに決定、代わりに見込みのある新人編集者向けホームページならびに新人向けタスクの両機能に集中する予定です. 試行結果の詳細はこちらをご参照ください.



現状

 * 2019-09-30: ヘルプパネルの試行を終了、要点を公表.
 * 2019-07-22: A/B 試験中、アラビア語版ウィキペディアに実装したヘルプパネル. 新人編集者の半数には非表示.
 * 2019-03-26: を公表.
 * 2019-03-04: 名前空間ごとに閲覧モードでヘルプパネルを表示させた. ヘルプ、ヘルプのトークページ、ウィキペディア、ウィキペディアのトークページ、利用者ページ、そのトークページ.
 * 2019-02-28: A/B 試験中、ベトナム語版ウィキペディアに実装したヘルプパネル. 新人編集者の半数には非表示.
 * 2019-02-14: チェコ語版と朝鮮語版ウィキペディアの被験者グループで実装した検索機能.
 * 2019-01-11: A/B 試験中、チェコ語版と朝鮮語版ウィキペディアに実装したヘルプパネル. 新人編集者の半数には非表示. 対象ウィキ群でヘルプパネルから質問を閲覧する方法は、それぞれのヘルプデスクを参照してください.
 * チェコ語版ヘルプデスク
 * 朝鮮語版ヘルプデスク
 * 次：利用者の行動をうながす多段階の機能を開発中で、実際に質問を投稿したり、検索結果の質を高めたり、またヘルプデスクの返信をちゃんと読む確率を上げることまで目指せたらと考えます.

実験の結果（2019年9月）

 * 新人利用者向けのヘルプパネルは2019年1月に初めて実装し、同年9月時点でそのデータ分析を終えて影響力の検討準備ができました. これは概要であり、より詳細な分析結果は公開の準備中です.
 * 概略としては、ヘルプパネルは活動（実際に編集作業をしたかどうか）あるいは新規利用者の定着率（編集のためにまたウィキを開いたかどうか）の増加を 示していません. 期待外れの結果ではありますが、今後も成果のありそうな機能を何種類か実験していくつもりです.
 * ヘルプパネルはごくわずかですが、実際に編集量（編集の回数）に変化を もたらしました が、結果とは衝突が見られます. 編集回数は新人編集者の初日こそ増えたものの、その後の2週間で減少しています.
 * 全般として、今のところはヘルプパネルがなぜこう言う結果を生んだか原因はまだ見つからず、仮説は複数あり今後の検討課題です. 一例として、ヘルプパネルは編集の仕方を解説しており、新規参加者が最初の編集を始める気持ちになるだろうと期待していました. 可能性として、かえってややこしい説明が多いと受け取られ、新人利用者に読むのがめんどくさいとか、難しそうと思わせて やる気を失わせた とも考えられます.
 * たしかに私たちのチームでは新人編集者の定着と活動量を増やすことが究極の目標であり、また今回の観察によると新人利用者はかなりの率でヘルプパネルを開き（20%）、開いた人は使っています（50%）. ここから、ヘルプパネルが提供する種類のヘルプの需要はあるとわかりました.
 * これらの結果を総合すると期待以下ではありました -- ヘルプパネルから指標にプラスの影響がきちんと出せると確信していたからです. ヘルプパネルの改善案はたくさんありますが、直近の課題としてこれ以上のトライと開発はしないこと、注力する課題は新人利用者向けのホームへならびに お勧め編集へシフトすることが決まりました. どちらも初期の結果に期待でき、ヘルプパネルよりもさらに多くの新規参加者を対象にしています. 新規参加者向けタスクのプロジェクトではヘルプパネルを活用し、今後とも Growth チームの取り組む編集初学者の経験に組みこむ予定です. 将来的に掘り下げた開発を再開する可能性はあります.

要約
It is common for new editors to face technical, conceptual, and cultural challenges when they start editing. They have specific questions, and could benefit from a human editor answering their questions. Most wikis have a help desk where new editors can ask questions, and such venues have been proven to be effective, especially the Teahouse in English Wikipedia (see this paper for an analysis of impact). The problem is that most new editors don't know about or find the help desk -- even though they frequently receive "welcome templates" on their talk pages with links.

仮説を2件、立てました.


 * Newcomers are more likely to seek help when presented with an easy way to submit questions.
 * Newcomers are more likely to complete edits by knowing where to seek help.

Therefore, the "Focus on help desk" project contains two parts:


 * 1) Invite new editors to the help desk.  We had four ideas for how to invite them: by posting on their talk pages, by using banners, through email, or by adding a link in the editing context.  We have decided to invite new editors to the help desk by adding a clear and obvious link to the help desk inside their editing experience, which we are calling the "help panel". That's for these reasons:
 * 2) * We know from the New Editor Experiences research that getting help in-context is important. This would offer a link to get help at the moment when users need it, and in a place that currently has no clear way to get help.
 * 3) * We have open questions around the "In-context questions or chat" idea, and this will help us learn whether there is demand for such a feature.
 * 4) Once we are successful at driving new editors to the help desk, undertake improvements to the help desk itself that make it a better place to ask questions.  This might include listing new questions at the top of the page or notifying new editors when their questions have been answered.

The Growth team built Part 1 (the help panel) first during 2018 Q2 (October 2018 - December 2018), and deployed it in Czech and Korean Wikipedias on January 11, 2019 inside a controlled A/B test. This experiment will take until July 2019 to fully complete. As results come in, we may add additional capabilities to the feature, deploy it in additional wikis, or decide to begin Part 2 (undertaking improvements to the help desk itself).

賛否のレビュー
Our team's designer reviewed the way that other platforms (e.g. SurveyMonkey, American Express, Codecademy) offer help in-context to their users (see for details). We think there are best practices we can learn from other software, especially when we see the same patterns across many different types of software. Even as we incorporate ideas from other software, we will still make sure to preserve Wikipedia's unique values of openness, clarity, and transparency. The comparisons are shown in this slide deck, and the main takeaways are:

In-context help UIs


 * In-context help is usually presented as a sticky button or tab in the bottom right of the screen
 * Clicking on help will usually open up a sticky panel over the UI offering one or more help options
 * Users are often able to search and view top help topics within the in-context help panel
 * In-context help will also link users to the dedicated/primary help center/forums
 * General/site-wide Help is usually a primary navigation option as well, and may also enable access to chat/feedback within the UI
 * Indicators of privacy and security also often shown in the expanded help UI
 * Mobile (app/web) versions typically do not have help accessible in-context but accessed as a primary navigation item that opens separately

そのコンテキストに直結するヘルプの種類


 * ヘルプセンター/ヘルプデスクへのリンク
 * よくある質問で、そのコンテキストに特化したやり取り
 * その場でチャット
 * 動画/チュートリアル
 * ガイド付きツアーのリンク
 * 過去のヘルプクエリの履歴
 * Community forums - used in a couple of places where there is less expectation of immediacy in response times and level of ‘service’
 * “Report a bug” - for users to send feedback about unexpected behaviors/bugs, often with the option to include a screenshot of the UI. Implicit expectation is the user should not to expect a response.

ヘルプ内にチャット/よくある質問の機能を


 * Users are able to email questions using the chat panel when there is no “Live” agent is available
 * Users are given feedback on response time windows and where they can expect answers
 * Voice and tone of chat are friendly and personable, sometimes with emoji and gif options in chat.
 * Users likely expect a level of service, as chat is typically provided by paid customer service reps.
 * Privacy links and information about how chat history is managed are clearly shown
 * Frequently asked questions/topics common for the particular context are often presented in the same UI as an less engagement-heavy help alternative for users
 * Automated answers / “Answer bots” are sometimes offered first to try to resolve the issue before users are given the option to post their question

ウィキメディアの同類の機能を評価
ヘルプパネルのアイデア出しをしている時、財団職員から過去に同類の機能が検討された話を聞きました. MoodBarArticle Feedbackは（どちらも現存せず）情報提供という意味でこちらの開発で参考になると考えます. その点はタスクで担当、おもな参考点は次のとおりです. 記事のフィードバック・ツール


 * アカウントの作成で編集者の活動を促す - 編集作業を手がける利用者が手助けを頼む投稿をしたとき、アカウントを作成すると「ヘルプチーム」からフィードバックが来ると伝え、登録を呼びかける方法があります.
 * 自由記述式の質問欄を作る前に、あらかじめヘルプの選択肢を増やしておく - 記事フィードバック・ツール Article Feedback Tool で得た知見によると、記事に対して関連性が低くあるいは構造化していないコメントを楽に投稿できることこそ、仲裁者に無用のノイズを増やすばかりです. 編集自体に関心があって特定のタスクに手助けを求める人なら不要な質問はしないと仮定はしていますが、ヘルプを自分で探せる道すじを用意すると発生件数をさらに減らせる可能性があります.
 * コメントに関連性を持たせる機構 – AFT ではコメントに関連性が欠けており、編集者の低評価の核でもあります. 関連性に基づくヘルプを実現するには、理想としては質問者やコメント者が自分はどの記事もしくは UI のどの部分を話題にしているか具体的に示す – たとえば質問に画面キャプチャを添えて投稿する、あるいはまた、文章のどこに対するコメントなのか対象部分を強調表示できると役立ちます.
 * 実際の編集に対話式に関与する - このツールには別の作業キューを想定し、経験を積んだ編集者が個人の「ウォッチリスト」や「最新の更新」では追いきれない対象、現状のプロセスではキュレーションできないが除去するべき有害なコンテンツを検知します. ここで重要な点は発生するトラフィック負荷の管理を既存システムで行うことで、いちばん簡単な対処法は、私たちが作る関与機能による相互作用をウィキ上の投稿として扱うことです.

MoodBar


 * Capturing email for responses is an effective option – As shown in the MoodBar experiment, email notifications is an effective means of engaging new editors.
 * Clear, prominent call to action for new editors to ask for feedback/help - per high level findings, the addition of a tooltip drawing attention to the MoodBar significantly increased its use.
 * Tracking the perceived helpfulness of responses is important - It would be useful for us to track similar new user perceptions of feedback, since a poor experience from receiving unhelpful advice has been shown to negatively impact editing.

Design for help panel
Our evolving designs can always be found in this interactive prototype. These clickable mockups also contain other design ideas for future iterations of the help panel, although some elements in those mockups may be obsolete. To see the latest version of the help panel, edit a page in Beta or Test wiki after having turned the help panel on in your preferences.

The "Comparative review" and "Review of similar MediaWiki features" were critical in the design process because they helped us "dream big" to explore the space of what the help panel could eventually become. Our team mocked up and discussed many ideas that probably will not become part of the help panel unless we see that earlier, simpler versions are successful. First, we are going to work on an initial version of the help panel to be deployed in January 2019.

Initial version
In the initial version of the help panel, we want to answer these questions:


 * Will newcomers click on a clear option to get help during the editing experience?
 * Do newcomers seem more interested in reading content to answer their own questions, or in asking a question for others to answer?
 * Will newcomers take action to write and post a question to the help desk?
 * Will the presence of the help panel increase new editor retention?

Therefore, we have decided to include these elements in the initial version, detailed in T206717 and T209318:


 * A call to action in the editing context.
 * A panel that opens containing links to helpful existing pages.
 * The ability to ask a question from that panel, and for that question to be automatically posted to the wiki's help desk.
 * The ability to add an email address to their account if the user does not already have one, and to modify their email address if it is not yet confirmed (and then to receive a confirmation email upon submitting).
 * The ability to turn off the help panel by clicking through to Preferences.

The mockups below show initial designs for that functionality as of November 2018 (to see the most up-to-date designs, use this interactive prototype). Right now, we will show this only to newcomers (meaning new account holders who are not auto-created). We also know that the wording in the feature is very important here, because it is critical that newcomers understand where and how information they write will be posted, and when and where they can except a reply. These mockups only contain some initial drafting of the wording, and we'll continue to refine it. We are hoping for strong ideas from community members, so please post any ideas on the talk page.

Business rules for initial version
Below are the current rules we're using to determine who receives the help panel and how it works. These may change as our team and the community continues to discuss and learn from the feature.


 * Users: the help panel will be a feature that can be turned on and off in user preferences. When it is deployed in January, it will be turned on to only newly created accounts (not auto-created) from the deployment date forward (except for a control group that will not have the feature).  All other users whose accounts were created before the deployment date will have the feature turned off, but will be able to go to their preferences to turn it on if they wish to try it out.  We will also be embedding an option to turn the help panel off inside the feature.  Our team discussed the possibility of turning the help panel on for all users, but wanted to first learn about how new users interact with it, and we want to make sure not to flood help desks with incoming questions.  See this Phabricator task for the discussion.
 * Namespaces: the help panel will be available in all namespaces. Our team discussed limiting the feature to only the main article namespace because the help documentation linked in the panel is mostly relevant to that namespace.  Ultimately, we decided to make it available in all namespaces so that we can learn in which namespaces people are most commonly looking for help -- we know that new editors struggle to edit the Talk namespace.  We will also encourage communities to develop documentation relevant to editing other namespaces (particularly the Talk namespace) so that we can make those links available in the appropriate namespaces.  See this Phabricator task for the discussion.
 * Headers: the help panel will automatically post users' questions to their wiki's help desk, and will automatically generate a header for that post. We have decided that the header will include the title of the page from which the question originates (with a link), as well as a timestamp to distinguish it from questions that may have originated from that some page.  Users will be able to decline to include the title in their header, if they consider it sensitive.  See this Phabricator task for the discussion.  Our preference is to number the posts instead of including a long and redundant timestamp, but that will require additional engineering work for a future version.

Wording for initial version
The wording for the help panel is complete, and the current state can be seen in the interactive mockups. Some of the most important things we have been considering in the wording are:


 * How to make it clear that by asking a question in the help panel, users will be posting in public?
 * How to encourage users to add their email address, since that is the best way for them to find out when they get a response to their question?
 * How to give users the option of excluding the title of the page they're editing from the automatically-generated header in the help desk, if they want to keep that private?
 * How to correctly set expectations about how quickly a user can expect a help desk response?
 * What to include in the automatically-generated header in the help desk?

Future versions
In determining the initial version of the help panel, many ideas for features were set aside to be evaluated later on. Many of them are visible in the initial mockups here. The following is a list of capabilities that are either being built for the help panel, or may be built in the future:


 * Nearer term
 * ✅ Search: the ability to search help pages. Users found this option appealing during testing, and it was built in T209301 and deployed on 2019-02-14.Help panel kowiki search 2019-02-15.png
 * ✅ More contexts: since help desks have not yet been overwhelmed with incoming questions, we are thinking of exposing the help panel to newcomers in contexts beyond editing -- perhaps while they are viewing pages in the Help or Wikipedia namespaces. We thought of this because the EditorJourney analysis showed that large portions of newcomers viewed pages in those namespaces during their first 24 hours.  Currently being built in T215664.
 * Greater affordance: so far, it looks like a minority of users who see the help panel button actually open it up. We could attempt to make it more visually obvious that it is a recommended place to start.
 * Vary links with context: the help panel contains five links to help pages, and about half of users that see them click on one. Different links are probably valuable in different contexts.  For instance, there might be some links for visual editor and some for wikitext.  Or some that are helpful on a talk page and some on an article page.  This is being worked on in T211117.
 * Longer term
 * Live chat: allowing newcomers to immediately chat with experienced editors right from the help panel. This would be a major project with many challenges.  See this page for our notes so far.
 * Suggested answers: many questions that users asked may have already been answered before, and we might be able to provide suggested answers as users type. This would be relevant once many more questions have been asked.  Ticketed in T209327.
 * Feedback on responses: newcomers may find the responses to their questions more or less satisfactory. By allowing newcomers to rate their responses, we may increase higher quality responses, and make it possible to surface the best ones to future newcomers.  Ticketed in T209332.
 * Marking question context: newcomers may have questions related to very specific elements of the editing experience. This idea would allow them to indicate exactly where on the page their issue is occurring.  Ticketed in T209328.
 * Templated replies: research shows us that the tone of a reply to a newcomer can make a difference in whether they edit again. We may be able to help people answering questions to format their replies in a nurturing way.  Ticketed in T209331.
 * Anonymous editors: though our team is focused on registered newcomers, it is possible to make the help panel present for anonymous editors, though it will be more difficult for them to find out that they have received a response. Ticketed in T209325.

User testing for help panel
During the week of November 26, 2018, we used usertesting.com to conduct eight tests of the help panel interactive prototype with internet users unaffiliated with the Wikimedia movement. Four tests were done on desktop and four were done on mobile. In these tests, respondents are compensated for trying out the mockups, speaking aloud on what they observe, and answering questions about the experience. As our team's designer described on the Phabricator task, the goals of this testing were:


 * 1) Gauge the discoverability of the help pane call to action and help pane
 * 2) Identify improvements to the usability of the help pane:
 * 3) * Do users understand they can edit and use help links at the same time?
 * 4) * Are users able to successfully follow the steps to post a question?
 * 5) * Is there anything users that would like to be able to do from the help pane that they are missing?
 * 6) Gauge user reactions to the help pane and expectations of how their questions will be answered.

Summary of findings


 * All users found the help panel contents to be very useful in getting help for editing.
 * No one had issues with having the article title included in the question.
 * Most participants would post a question to the help desk as a last resort after trying to help themselves through clicking on links, with two people saying they would likely not post at all as they would want a more immediate response.
 * One user said they would more likely just Google the editing help over the options shown.
 * Whilst it was understood that the Help desk would be answered by volunteers, many participants wanted more reiteration and clarity of expected wait times.
 * Clear layout of options, shown in the order in which almost participants said they would seek for help.
 * Discoverability of the help panel call-to-action may be reduced in the case when the user is using the Visual Editor or wikitext2017 editor because those editors have a competing “?” icon.
 * One user suggested having an example question in hint text.
 * One user wanted more information shown about when their question was viewed and by how many people.
 * Two users mentioned the help recommendations shown on the Help Desk seemed disjointed from what was shown in the help panel.

Recommendations (not all of these will be implemented)


 * Prioritize the Searching help task inside the help panel.
 * Provide clearer times on both the review and confirmation screens for when a newcomer should expect a response from the help desk.
 * Add a more specific help question sample in the placeholder/hint text.
 * Experiment or A/B test alternative text on the initial “Ask a question” call-to-action.
 * Show a first-run tooltip/pulse indicator to highlight the new help panel.
 * Add an additional menu item to existing editing toolbar help on Desktop VE or Wikitext2017 which opens the help panel.

Measurement and experiment plans
High-level questions

These are the main things we want to find out from the help panel.


 * 1) Does the presence of the help panel lead to questions being asked and answered?
 * 2) Does the presence of the help panel increase editor activation (making their first edit)?
 * 3) Does the presence of the help panel increase editor retention (coming back to edit again)?
 * 4) Does the presence of the help panel increase the proportion of constructive edits?

Controlled experiments

See this page for the detailed experiment plan.

In order to understand the help panel’s impact on editor activation and retention, we propose a six month A/B test. During that test, 50% of new registrations on target wikis will have the help panel enabled by default, and 50% will have it disabled. We anticipate that we will be able to detect 10% changes in activation after about one month, and 10% changes in retention after about six months. We are likely to be running other experiments on the target wikis at the same time, for example, testing variants of our welcome survey. Therefore, assignment of users to treatment and control groups for all tests will have to be coordinated so that we ensure the distributions are not biased.

Specific measurements: quantitative

''These are specific measurements that help us answer the high-level questions. These are measurements we will be able to do programmatically. See this EventLogging schema for the exact details on the data that is being recorded and stored.''


 * 1) What is the context the user is in when they start interacting with the help panel?
 * 2) What percent of users open the help panel?
 * 3) What percent close the help panel without using any part of it?
 * 4) What percent click one or more of the help links?
 * 5) How far do users go in the workflow of asking a question?
 * 6) What percent of users turn the help panel off?
 * 7) What is the average length of questions asked via the help panel?
 * 8) What percent of users click any of the links in the final confirmation page of asking a question?
 * 9) What percent of users who post a question return to view the answer?
 * 10) Does interacting with the help panel alter the probability of abandoning an edit session?
 * 11) What percent of users use the help panel more than once?
 * 12) Are newly registered users with an email address likely to return to view the answer to their question?
 * 13) What percent of newcomers that had no email address add one when asking a question?
 * 14) What percent of newcomers confirm their email address (within 48 hours) after asking a question?
 * 15) What percent of newcomers ask a question without an email address?
 * 16) What percent of newcomers submit a question without including the title of the page they were editing?
 * 17) How does help panel usage vary with welcome survey responses?

Specific measurements: qualitative

''These are specific measurements that help us answer the high-level questions. These are measurements we will need to do manually, by counting and reading the contents of help desks. Doing these programmatically would be an inordinate amount of work.''


 * 1) What percent of questions get answered?
 * 2) How quickly do questions get answered?
 * 3) How often do newcomers edit the question they asked?
 * 4) What kind of questions do people ask and which ones are more likely to get answered?

Usage counts
After about two months of being deployed to half of new accounts in Czech and Korean Wikipedias, and two weeks of being deployed to half of new accounts in Vietnamese Wikipedia, these are the numbers (as of 2019-03-28):


 * 3,343 users have seen the help panel button
 * 566 users have opened up the panel
 * 225 users have clicked a link in the panel
 * 51 have run searches
 * 51 users have submitted a question to the help desk (Czech: 34, Korean: 8, Vietnamese: 9)

Leading indicators
The help panel's experiment plan defines a set of leading indicators that the team evaluates in order to determine whether the feature is behaving as expected, and whether there are any urgent problems that need to be solved.

Using a month of data, we published an initial evaluation of leading indicators here. In summary, the help panel seems to be behaving in a healthy way, with good numbers of newcomers opening and using it. The analysis identified a couple areas for improvement that the team has taken action on:


 * Because we wanted to increase the number of newcomers who open the panel, we started showing it in additional contexts on March 4, 2019. Newcomers now see the help panel when reading in the Wikipedia, Help, and User namespaces.
 * Because we saw that in Korean Wikipedia very few users ask questions, we built the search feature so that newcomers would have an easier way to find helpful information on their own.

Experiment results
See results above.

Resources

 * More on this feature: The help panel contains a link to learn "More about this feature". That link will lead to this page, which will hopefully answer user questions about how the feature works.  Feel free to translate it into your language!
 * Best practices: Because many communities don't have a lot of experience concerning help desks, the WMF Community Relations Team has assembled on a page with some best practices that can help communities have more successful collaborations with newcomers. Please feel free to translate that page into your language!