# Growth/Personalized first day/構造化したタスク/リンクの追加

Other languages:

このページでは Growth チームが手がける構造化タスク「リンクを追加」について説明しており、Growth チームはこれを構造化したタスクのひとつとして新規参加者のホームページに載せています。 このページでは主要なアセット、設計、未決の課題、意思決定について述べます。 進捗状況で増えた更新のほとんどは一般向けのGrowthチームの更新ページに、このページには特定の大規模または詳細な更新をそれぞれ掲載します。

2021年8月時点でこのタスクの反復開発の初回をウィキペディアの新規登録アカウントの半数に実装することにして、以下の言語版で実施しました。アラビア語版、チェコ語版、ベトナム語版、ブラジル語版、ベンガリー語版、ポーランド語版、フランス語版、ロシア語版、 ルーマニア語版、ハンガリー語版、ペルシャ語版。 機能の実装から 2週間に集計したデータを解析したところ、新規参加者はこれらの編集をたくさんこなしており、その上、巻き戻し率は低いとわかりました。 この分析からわかった事柄は機能の改善に活用され、成果が現れてもっと多くのウィキに実装するまでになりました。

インタラクティブな試作版をご用意しましたので、開発の対象をご覧ください。 ただし試作版のため、押しても動かないボタンもあります。

チームからウィキマニア2021において、前提情報とアルゴリズム、実装と成果を発表しました。 動画はこちら。 and the スライド集はこちら。

## 現状

• 2020-01-07: リンク候補のアルゴリズム の有効性、初回の評価
• 2020-02-24: 改良したリンク候補のアルゴリズムを評価
• 2020-05-11: コミュニティと協議、構造化したタスクとリンク候補
• 2020-05-29: 基本的な枠組みの検討
• 2020-08-27: バックエンドの開発作業に着手
• 2020-09-07: モバイル版設計、初回のユーザーテスト
• 2020-09-08: 設計最新版について、コミュニティに協議を呼びかける
• 2020-10-19: モバイル版設計、第2回のユーザーテスト
• 2020-10-21: デスクトップ版設計、初回のユーザーテスト
• 2020-10-29: フロントエンドの開発作業に着手
• 2020-11-02: デスクトップ版設計、第2回のユーザーテスト
• 2020-11-10: 設計へのコミュニティのフィードバック募集、アラビア語、ベトナム語、チェコ語
• 2021-04-19: 用語集と数値に節を追加
• 2021-05-10: 先行ウィキ4件で機能をテスト
• 2021-05-27: 新規参加者の半数に実装、ウィキペディアのアラビア語版、ベトナム語版、チェコ語版、ベンガリー語版
• 2021-07-21: 新規参加者の半数に実装、アラビア語版、ウィキペディアのポーランド語版、ロシア語版、フランス語版、ルーマニア語版、ハンガリー語版、ペルシャ語版
• 2021-07-23: 機能実装から最初の2週間における分析を公表。
• 2021-08-15: ウィキマニアで発表、前提条件と実装、アルゴリズム、成果を説明。
• : マイナーな改善、より多くのウィキに実装予定、解析をより深める。

## 要約

• Are structured tasks engaging to newcomers?
• Do newcomers succeed with structured tasks on mobile?
• Do they generate valuable edits?
• Do they lead some newcomers to increase their involvement?

The below is excerpted from the structured tasks page, explaining why we chose to build "add a link" as the first structured task.

The Growth team currently (May 2020) wants to prioritize the "add a link" workflow over the other ones listed in the table above. Although other workflows, such as "copyedit", seem to be more valuable, there are a set of reasons we would want to start first with "add a link":

• In the near term, the most important thing we would want to do first is to prove the concept that "structured tasks" can work. Therefore, we would want to build the simplest one, so that we can deploy to users and gain learnings, without having to invest too much in the first version. If the first version goes well, then we would have the confidence to invest in types of tasks that are more difficult to build.
• 私たちの側からは、最も簡単にビルドできそうなのが「リンクを追加」であり、WMF 研究チームが作成したアルゴリズムがすでにあること、それを使うとウィキリンク作りのおすすめが円滑に進むと予測したからです (アルゴリズムの節をご参照ください。)
• Adding a wikilink doesn't usually require the newcomer to type anything of their own, which we think will make it particularly simple for us to design and build -- and for the newcomer to accomplish.
• Adding a wikilink seems to be a low-risk edit. In other words, the content of an article can't be as compromised through adding links incorrectly as it could through adding references or images incorrectly.

## デザイン

This section contains our current design thinking. To look into the full set of thinking around designs for the "add a link" structured task, see this slideshow, which contains background, user stories, and initial design concepts.

Our designs evolved through several rounds of user tests and iterations. この機能の初回のバージョンとしては、2020年12月時点で設計案がまとまり開発に取りかかることになりました。 You can see them in these interactive prototypes. Note that because they are prototypes, not all buttons work:

### Comparative review

• Task types – are divided into five main types: Creating, Rating, Translating,  Verifying content created by others (human or machine), and Fixing content created by others.
• Visual design & layout – incentivizing features (stats, leaderboards, etc) and onboarding is often very visually rich, compared to pared back, simple forms to complete short edits. Gratifying animations often compensate for lack of actual reward.
• Incentives – Most products offered intangible incentives grouped into: Awards and ranking (badges) for achieving set milestones, Personal pride and gratification (stats), or Unlocking features (access rights)
• Users motivations – those with more altruistic motivations (e.g., help others learn) are more likely to be incentivized by intangible incentives than those with self-interested motivations (e.g., career/financial benefits)
• Personalization/Customization – was used in some way on most apps reviewed. The most common customization was via surveys during account creation or before a task; and geolocalization used for system-based personalization.
• Guidance – Almost all products reviewed had at least basic guidance prior to task completion, most commonly introductory ‘tours’. In-context help was also provided in the form of instructional copy, tooltips, step-by-step flows,  as well as offering feedback mechanisms (ask questions, submit feedback)

### Initial wireframes

After organizing our thoughts and doing background research, the first visuals in the design process are "wireframes". These are simply meant to experiment and display some of the ideas we think could work well in a structured task workflow. For full context around these wireframes, see the design brief slideshow.

### モバイルの試作版：2020年8月

Our team discussed the wireframes from the previous section. We considered what would be best for the newcomers, taking into account the preferences expressed by community members, and thinking about engineering constraints. 試作版作りの次の段階に進んだ2020年8月、提案した機能がより詳細に把握できるものを目指しました。 These mockups (or similar versions) will be used in team discussions, community discussions, and user tests. One of the most important things we thought about with these mockups is the concern we heard consistently from community members during the discussion: structured tasks may be a good way to introduce newcomers to editing, but we also want to make sure they can find and use the traditional editing interfaces if they are interested.

We have mockups for two different design concepts. We're not necessarily aiming to choose one design concept or the other. Rather, the two concepts are meant to demonstrate different approaches. 設計の最終案は、両方のコンセプトから良いところを取り入れるかもしれません。

• Concept B: the structured task edit takes place in its own new area. The user is shown only the paragraph of the article that needs their attention, and can go edit the article if they choose. Fewer distractions from adding links, but more distant access to the visual and source editors.

Please note that the focus in this set of mockups is on the user flow and experience, not on the words and language. 今後、機能に書く文はどうするか、また利用者にリンクを追加するべきかどうか説明するため、最善の方法を検討します。

Static mockups

To view these design concepts, we recommend viewing the full set of slides below.

Interactive prototypes

You can also try out the "interactive prototypes" that we're using for live user tests. These prototypes, for Concept A and for Concept B, show what it might feel like to use "add a link" on mobile. They work on desktop browsers and Android devices, but not iPhones. Note that not everything is clickable -- only the parts of the design that are important for the workflow.

これらの設計を議論する時、私たちのチームでは、いくつかの根源的な質問に回答を得たいと望んでいます。

1. 編集は記事空間で実行しますか？ (詳細のツメが必要)  あるいは、こういう種類の編集体験用に別途、設定しますか？ (目的に最適化してあるだけに、実際のエディタへ移行する段階でパワーが必要)
2. もしも、リンクのターゲットまたは文章の編集をしたい人が現れたら？ できない処理をしておくのか、通常のエディタへ誘導するのか？  ビジュアルエディタについて指導するタイミングと考えますか？
3. 新規参加者に従来の編集ツールを見つけてもらうには、サポートが肝心です。 では、そのタイミングは？ 構造化したタスクを体験している期間に、エディタを使うようにお知らせを表示する方法が良いですか？ それとも、達成度のマイルストンを利用して、たとえば特定の数の構造化タスクをこなしたタイミングで？
4. ここで「ボット」という単語は適切でしょうか？ 言い換えは可能でしょうか？ ひょっとすると「アルゴリズム」「コンピュータ」「機械」「自動◯◯」などは使えそう？ 機械がお勧めするものには間違いがあり、人間が考える重要性を伝えるには 、表現を改善できませんか？

### Mobile user testing: September 2020

Background

During the week of September 7, 2020, we used usertesting.com to conduct 10 tests of the mobile interactive prototypes, 5 tests each of Concepts A and B, all in English. By comparing how users interact with the two different approaches at this early stage, we wanted to better understand whether one or the other is better at providing users with good understanding and ability to successfully complete structured tasks, and to set them up for other kinds of editing afterward. Specific questions we wanted to answer were:

• Do users understand how they are improving an article by adding wikilinks?
• Do users seem like they will want to cruise through a feed of link edits?
• Do users understand that they're being given algorithmic suggestions?
• Do users make better considerations on machine-suggested links when they have the full context of the article (like in Concept A)?
• Do users complete tasks more confidently and quickly in a focused UI (like in Concept B)?
• Do users feel like they can progress to other, non-structured tasks?

Key findings

• The users generally were able to exhibit good judgment for adding links. They understood that AI is fallible and that they have to think critically about the suggestions.
• While general understanding of what the task would be ("adding links") was low at first, they understood it well once they actually started doing the task. Understanding in Concept B was marginally lower.
• Concept B was not better at providing focus. The isolation of excerpts in many cases was mistaken for the whole article. There were also many misunderstandings in Concept B about whether the user would be seeing more suggestions for the same term, for the same article, or for different articles.
• Concept A better conveyed expectations on task length than Concept B. But the additional context of a whole article did not appear to be the primary factor of why.
• As participants proceed through several tasks, they become more focused on the specific link text and destination, and less on the article context. This seemed like it could lead to users making weak decisions, and this is a design challenge. This was true for both Concepts A and B.
• Almost every user intuitively knew they could exit from the suggestions and edit the article themselves by tapping the edit pencil.
• All users liked the option to view their edits once they finished, either to verify or admire them.
• “AI” was well understood as a concept and term. People knew the link suggestions came from AI, and generally preferred that term over other suggestions. This does not mean that the term will translate well to other languages.
• Copy and onboarding needs to be succinct and accessible in multiple points. Reading our instructions is important, but users tended not to read closely. This is a design challenge.

Outcome

• We want to build Concept A for mobile, but absorbing some of the best parts of Concept B's design. These are the reasons why:
• User tests did not show advantages to Concept B.
• Concept A gives more exposure to rest of editing experience.
• Concept A will be more easily adapted to an “entry point in reading experience”: in addition to users being able to find tasks in a feed on their homepage, perhaps we could let them check to see if suggestions are available on articles as they read them.
• Concept A was generally preferred by community members who commented on the designs, with the reason being that it seemed like it would help users understand how editing works in a broader sense.
• We still need to design and test for desktop.

Ideas

The team had these ideas from watching the user tests:

• Should we consider a “sandbox” version of the feature that lets users do a dry run through an article for which we know the “right” and “wrong” answers, and can then teach them along the way?
• Where and when should we put the clear door toward other kinds of editing?  Should we have an explicit moment at the end of the flow that actively invites them copyedit or do another level task?
• It’s hard to explain the rules of adding a link before they try the task, because they don't have context. How might we show them the task a little bit, before they read the rules?
• Perhaps we could onboard the users in stages?  First they learn a few of the rules, then they do some links, then we teach them a few more pointers, then they do more links?
• Should users have a cooling-off period after doing lots of suggestions really fast, where we wait for patrollers to catch up, so we can see if the user has been reverted?

### Desktop mockups: October 2020

After designing, testing, and deciding on Concept A for mobile users, we moved on to thinking about desktop users. We again have the same question around Concepts A and B. The links below open interactive prototypes of each, which we are using for user testing.

• Concept A: the structured task takes place at the article, in the editor, using some of the existing visual editor components. This gives users greater exposure to the editing context and may make it more likely that they explore other kinds of editing tasks.
• Concept B: the structured task takes place on the newcomer homepage, essentially embedding the compact mobile experience into the page. Because the user doesn't have to leave the page, this may encourage them to complete more edits. They could also see their impact statistics increase as they edit.

We are user testing these designs during the week of October 23. See below for mockups showing the main interaction in each concept.

Outcome

The results of the desktop user tests led us to decide on Concept A for desktop for many of the same reasons we chose Concept A for mobile. The convenience and speed of Concept B did not outweigh the opportunity for Concept A to expose newcomers to more of the editing experience.

### Terminology

"Add a link" is a feature in which human users interact with an algorithm. As such, it is important that user have a strong understanding that suggestions come from an algorithm and that they should be regarded with skepticism. In other words, we want the users to understand that their role is to evaluate the algorithm's suggestion and not to trust it to much. Terminology (i.e. the words we use to describe the algorithm) play an important role in building that understanding.

At first, we planned to use the terms "artificial intelligence" and "AI" to refer to the algorithm, but we eventually decided to use the term "machine". This may be a practice that gets adopted more broadly as multiple teams build more structured tasks that are backed by algorithms. Below is how we thought about this decision.

Background

As we build experiences that incorporate augmentation, we are thinking about the terminology to use when referring to suggestions that come from automated systems. If possible, we want to make a smart choice at the outset, to minimize changes and confusion later. For instance, we are looking at sentences in the feature like these:

• "Suggested links are machine-generated, and can be incorrect."
• "Links are recommended by machine, and you will decide whether to add them to the article."

Objectives

We want the terms we use to satisfy these objectives.

• Transparency: users should understand where recommendations come from, and we should be being honest with them.
• Human-in-the-loop: users should understand that their contributions improve recommendations in the future.
• Usability: copy should help users complete the tasks, not confuse or burden them with too much information.
• Consistency: we should use the same copy as much as possible to lower cognitive load.

Terms we considered

 Term Strengths Weaknesses AI Well understood in some cultures. Users naturally understand that it can fail. May not accurately describe all the various augmentation we use. Users may think methods are more sophisticated or opaque than they are. Can be challenging to translate. Machine Moderately understood. Wide applicability. Some users may not understand what we mean / term seems archaic. Unclear how much to trust it. Computer Moderately understood. Wide applicability. Sounds archaic Algorithm Well understood. Wide applicability. Unclear how much to trust it. Different meaning in ML context. Automated Total applicability. So broad that it may not convey much meaning to users.

Methods and findings

• User testing: the Growth team tested "add a link" in English using the terms "artificial intelligence" and "AI". We found that users understood the term well and that English-speaking users understood that they should regard the output of AI with skepticism.
• Experts: we spoke to WMF experts in the fields of artificial intelligence and machine learning. They explained that the link recommendation is not truly "AI", in the way that the term is used in the industry today. They explained that by using that term, we may be over-inflating our work and giving users a false sense of the intelligence of the algorithm. Experts preferred the term "machine", as it would accurately describe the link recommendation algorithm as well as be broad enough to describe almost any other kind of algorithm we might use for structured tasks.
• Multi-lingual community members: we spoke to about seven multilingual colleagues and ambassadors about the terms that would make the most sense in their languages. Not all languages have a short acronym for "AI"; many have long translations. The consensus was that "machine" made good sense in most languages and would be easy to translate.

Result

We are going to use the term "machine" to refer to the link recommendation algorithm, e.g. "Suggested links are machine-generated". See screenshots below to see one of the places where the terminology changed based on this decision.

## Measurement

### Hypotheses

The “add a link” workflow structures the process of adding wikilinks to a Wikipedia article, and assists the user with artificial intelligence to point out the clearest opportunities for adding links.  Our hypothesis with the “add a link” workflow is that such a structured editing experience will lower the barrier to entry and thereby engage more newcomers, and more kinds of newcomers than an unstructured experience.  We further hypothesize that newcomers with the workflow will complete more edits in their first session, and be more likely to return to complete more.

Below are the specific hypotheses we seek to validate. These govern the specifics around which data we'll collect and how we'll analyze it.

2. “Add a link” edits are more likely to be successful than unstructured suggested edits, meaning that a user completes the task and saves the edit. They are also more likely to be constructive, meaning that the edit was not reverted, than unstructured suggested edits.
5. The perceived quality of the link recommendation algorithm will be high, both from the users who make "add a link" edits and the communities who review those edits.
6. Users who get “add a link” and who primarily use/edit wikis on mobile see a larger increase in the effects on retention and productivity relative to desktop users.

### Experiment

We plan to continue to have a Control group that does not get access to the Growth features, which is a randomly selected 20% of new registrations.

• Group C: control (no Growth features)

The experiment will run for a limited time, most likely between four to eight weeks. In practice the experiment will start with our four pilot wikis.  After two weeks, we will analyze the leading indicators below to decide whether to extend the experiment to the rest of the Growth wikis.

### Leading Indicators and Plan of Action

We are at this point fairly certain that Growth features are not detrimental to the wiki communities. That being said, we also want to be careful when experimenting with new features. Therefore, we define a set of leading indicators that we will keep track of during the early stages of the experiment. Each leading indicator comes with a plan of action in case the defined threshold is reached, so that the team knows what to do.

Indicator Plan of Action
Revert rate This suggests that the community finds the Add a Link edits to be unconstructive. If the revert rate for Add a Link is significantly higher than that of unstructured link tasks, we will analyze the reverts in order to understand what causes this increase, then adjust the task in order to reduce the likelihood of edits being reverted.
User rejection rate This can indicate that we are suggesting a lot of links that are not good matches. If the rejection rate is above 30%, we will QA the link recommendation algorithm and adjust thresholds or make changes to improve the quality of the recommendations.
Task completion rate This might indicate that there’s an issue with the editing workflow. If the proportion of users who start the Add a Link task and complete it is lower than 75%, we investigate where in the workflow users leave and deploy design changes to enable them to continue.

#### Analysis and Findings

We collected data on usage of Add a Link from deployment on May 27, 2021 until June 14, 2021. This dataset excluded known test accounts, and does not contain data from users who block event logging (e.g. through their ad blocker).

Revert rate: We use edit tags to identify edits and reverts, and reverts have to be done within 48 hours of the edit. The latter is in line with common practices for reverts.

User registration Task type N edits N reverts Revert rate
Unstructured 63 22 34.9%

For the post-deployment group, a Chi-squared test of proportions finds the difference in revert rate to be statistically significant (${\displaystyle \chi ^{2}=16.5,df=1,p\ll 0.001}$). This means that the threshold described in the leading indicator table is not met.

Rejection rate: We define an "edit session" as reaching the edit summary or skip all dialogue, at which point we count the number of links that were accepted, rejected, or skipped. Users can reach this dialogue multiple times, because we think that choosing to go back and review links again is a reasonable choice.

User registration N accepted % N rejected % N skipped % N total
Post-deployment 597 72.4 125 15.2 103 12.5 825
Pre-deployment 1,464 65.1 595 26.5 189 8.4 2,248
2,061 67.1 720 23.4 292 9.5 3,073

The threshold in the leading indicator table was a rejection rate of 30%, and this threshold has not been met.

Over-acceptance rate: This was not part of the original leading indicators, but we ended up checking for it as well in order to understand whether users were clicking "accept" on all the links and saving those edits. We reuse the concept of an "edit session" from the rejection rate analysis, and count the number of users who only have sessions where they accepted all links. In order to understand whether these users make many edits, we measure this for all users as well as for those with five or more edit sessions. In the table below, the "N total" column shows the total number of users with that number of edit sessions, and "N accepted all" the number of users who only have edit sessions where they accepted all suggested links.

User registration N total N accepted all %
Post-deployment ≥1 edit 96 31 32.3
≥5 edits 6 0 0.0
Pre-deployment ≥1 edit 64 10 15.6
≥5 edits 19 1 5.3

We find that some users only have sessions where they accepted all links, but these users do not typically continue to make Add a Link edits. Instead, users who make additional edits start rejecting or skipping links as needed.

Task completion rate: We define "starting a task" as having an impression of "machine suggestions mode". In other words, the user is loading the editor with an Add a Link task. "Completing a task" is defined as clicking to save an edit, or confirming that all suggested links were skipped.

Post-deployment 178 96 53.9
Pre-deployment 101 64 63.4
279 160 57.3

The threshold defined in the leading indicator table is "lower than 75%", and this threshold has been met. In this case, we're planning to do follow-up analysis to understand more about the tasks, e.g. if they had a low number of suggested links, or if this happens on specific wikis or platforms.