Reading/Web/コンテンツ検索エクスペリメント/簡素な記事要約
|
簡素な記事要約
|
We would like to provide article summaries, which would simplify the content of the articles. This will make content more readable and accessible, and thus easier to discover and learn from. This part of the project focuses only on displaying the summaries. A future experiment will study ways of editing and adjusting this content.
Previous research and experiments
これまでのリサーチ
今のところ百科事典レベルのコンテンツの多くは長いため迅速な把握は困難です。 さらにこれら記事は平均的な成人の読解レベルよりもはるかに高いレベルで書かれています。 Simple English Wikipedia や Basque Txikipediaなどのコンテンツを簡素化するプロジェクトはそういった問題の一部に対処するように設計されています。 編集者が記事のより簡素なバージョンを手動で作成させているのです。 しかしこのようなプロジェクトは部分的に成功していますが数言語のみでの提供です。大規模化が困難でした。 さらに編集者はすでに書いた内容を書き直すように依頼されることもあります。 This can feel very repetitive.
前回の調査（コンテンツの簡素化）において次の2つのニーズがわかりました。
- The need for readers to quickly get an overview of a given article or page
- The need for this overview to be written in language the reader can understand
Wikimania 2024 セッション
ウィキマニア2024においてウィキメディアンは、AI/機械生成による既存コンテンツのリミックスを活用しウィキペディアをよりアクセスしやすく、学習しやすいものにする方法について議論しました。このセッションでは編集者がこういった新機能の構造とコンテンツを制御できるようにする方法に焦点を当てました。
Results summary
- Participant editors thought that the features presented had potential to improve the quality of the learning experience on Wikipedia
- Participants agreed that some community/editorial control over machine generated content and suggestions would be preferable
- Participants flagged a preference for solutions which did not require a lot of upfront work from editors
- For most groups, this meant adding the ability to edit or otherwise flag issues with machine-generated/remixed content once it was published or generated automatically on the wikis
- Participants flagged a need for clearly marking which features AI was directly involved in
Userlytics experiment
ウェブ読者に記事の要約を提示する実験を行いました。クリックスルー率と利用パターンを指標とし、記事の要約が読者の定着率を向上させるかどうかを検証しました。すなわち提案されたコンセプトがユーザーにとって明確であり機能が実際に使えるかどうかを確認しました。
私たちは8人の参加者を対象にシンプル要約機能のプロトタイプに関するモデレートされていないUserlytics調査を実施しました。 The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions.
Results summary
Participants found simple summaries easy to use, useful, and had a high level of trust in the machine-generated summary. Responses were much more positive than expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed. See more at Usability study on Simple Summaries.
Browser extension experiment
The second experiment added the new feature into the pre-existing browser extension. The goal was to determine whether summaries are worth future investment for more costly experimentation. This time, we gathered evidence for the following hypotheses:
- Users are interested in the feature as presented (as measured through initial engagement with summary feature)
- Results: out of all pageviews where summaries were displayed, 8.09% of pageviews had a summary opened. The pages containing the summary received 10,198 impressions and 825 total clicks to open the summary.
- Users report positive experiences with the user experience of the feature (as measured by the answers to the "was this useful?" question)
- Results: 101 responses were received in our survey. 75.2% of all clicks selected "yes", 24.8% of all clicks selected "no".
The above summary is based on data collected from the English Wikipedia (enwiki) desktop platform between December 4 and 17, 2024. Only users who had enabled the extension could view the summary, and not all pages included one. See phab:T381253 for more details
Design
For the browser extension experiment, the designs below were used. Summaries appear as collapsed when the page is loaded. If a user is interested in opening the summary, the can click the dropdown arrow, which allows the summary to appear.
The dropdown opens the summary, as well as additional information on how the summary was generated. It also contains the "unverified" tag to indicate this summary has not been verified by editors. (Once a future moderation workflow is in place, this tag will show as "verified" after editor verification)
-
Screenshot of collapsed state of article summary browser extension experiment -
Screenshot of the open state of the simple article summary browser extension for the Dopamine article
次のステップ
Goals for further experimentation
Based on the results above, we believe this feature has the potential to significantly benefit readers. We have identified two hypotheses that will help guide and direct this work in the future that can focus both on the technical and moderation aspects of our next steps.
- Evaluate impact of a proposed summary feature on key user metrics on a Wikimedia wiki
- Evaluate community interest and sentiment towards the summary feature, and its range across different language communities
- Explore community involvement in the feature, specifically focusing on the need and approach towards summary moderation
- Explore requirements for improvements and applications of the model used for summary generation
モバイル要約パイロット（仮説1）
実稼働wikiのモバイルサイトにオプトイン機能として事前生成された要約機能を導入すると、4%を超えるクリックスルー率を測定でき、セッションの長さ、ページビュー、内部参照に悪影響が及ばないことを保証し、このデータを使用して要約機能をさらに拡張するかどうか、またどのように拡張するかを決定できるようになります。
Mobile pilot
To evaluate this hypothesis, we are building a pilot version of the feature using the pre-generated summaries evaluated in the browser extension experiment. This version will be available for opt-in to a small percentage of readers on the mobile website for an experimental period of two weeks in June 2025.
モバイル要約パイロット（仮説 2）
アンケートやウィキ内での議論などを通じコミュニティと協力して要約モデレーションの設計に取り組むと、機能の初期拡張に必要な最小限の実行可能なモデレーションワークフローを決定し、モデレーションをコミュニティ主導にするか、自動化するか (プロンプトレベルで)、またはその両方の組み合わせにするかを明確にすることができます。
Community surveys
To evaluate this hypothesis, we are building a survey for communities. This survey will run in English, French, Japanese, and Spanish Wikipedias. The survey will present different options for adding, removing, and patrolling simple article summaries. We will use the results of the survey to propose the first version of a summary moderation workflow, which we will bring to communities for further discussion.
Timeline
| Estimated time | Planned activity | Next steps |
|---|---|---|
| 2023-2024 | Research on simplification | Start discussions around simple article summaries |
| August 2024 | Wikimania brainstorming session | Proceed with user study |
| October 2024 | Usability study | Proceed with browser extension experiment |
| December 2024 | Browser extension experiment | Proceed with experimentation on mobile
Begin discussions for moderation workflows with communities |
| June 2024 | Mobile pilot experiment | TBD |
| June 2024 | Community moderation and patrolling survey | TBD |