Talk pages project/Replying

This page talks about the Editing Team's work to improve contributors' workflows for replying to comments on talk pages, across Wikipedia's 16 talk namespaces. This new workflow for replying to specific comments is intended to make participating productively on talk pages easier and more intuitive for Junior and Senior Contributors.

This initiative sits within our team's larger effort to help contributors work together more effectively. To accomplish this, we will build upon existing community conventions to evolve talk pages. It is our intention to evolve talk pages in a way that gives experienced contributors more leverage to coordinate their work and connect with other editors, while making communicating on-wiki more accessible and intuitive for newer contributors.

To participate in and follow this project's development, we recommend adding this page to your watchlist. We will use this page to do things like:


 * Share potential designs and ask for feedback about them
 * Test prototypes
 * Post updates about this feature's development
 * Ask and invite questions about any part of this project (e.g., measurement, deployments, etc.)

Status updates
This section contains updates about the project's development.

17 December 2020
A/B Test

We have decided the key metric we will use to compare the A/B test's  and   groups will be the rate at which people in either group publish the comments they initiate. More information in can be found in the Metrics section below and in Phabricator here: T252057.

2 December 2020
Measuring engagement

The team is analyzing the Reply Tool usage data from the Arabic, Czech and Hungarian Wikipedias, where the tool has been available as an opt-out feature since 24-September-2020.

This analysis will help us determine whether people, across experience levels, are having success using the Reply Tool and whether people are using the tool in ways that disrupts others.

We will post the results of this analysis in the Metrics section below (see: "Analysis 2: Engagement"). You can expect these results to be posted no later than January, 2021.

Evaluating impact

The team is preparing to run an A/B test to evaluate the impact the Reply Tool has had on how people use talk pages. This test will help the team decide whether we should move forward with plans to offer the tool to all people, at all wikis, as an opt-out user preference.

More details about the test can be found in the Metrics section below (see: "Analysis 3: Impact").

4 November 2020
Deployment planning

Today, 4-November, the Reply Tool became available as an opt-in Beta Feature at an additional ~250 Wikipedias. This means the Reply Tool is now available as either a Beta Feature or opt-out user preference at all Wikipedias except for the following projects: English, Finnish, Gan, German, Inuktitut, Kazakh, Kurdish, Russian, Tajik, and Uzbek.

28 October 2020
Deployment planning

In the coming weeks, we are planning to offer the Reply Tool as an opt-in Beta Feature at all remaining Wikipedias except for the following projects: English, Gan, German, Inuktitut, Kazakh, Kurdish, Russian, Tajik, and Uzbek.

An exact date has not yet been set. Once it has, we will announce it on this page and in Tech/News.

''*Note: the above was also mentioned in this week's Tech/News. See: Tech/News/2020/44.''

16 October 2020
Deployment

On Wednesday, 14-October, the Reply Tool became available as an opt-in Beta Feature at an additional 30 projects. The objective of this deployment is to ensure the Reply Tool is usable and useful for editors at Wikipedias that write in languages/scripts with unique characteristics (e.g. long words, unusual font sizing, many diacrtics, rare fonts, etc.).

The full list of projects where the Reply Tool is currently available can be found here: Talk pages project.

25 September 2020
Deployment

On Thursday, 24-September, the Reply Tool became available to all users (logged in and out) at the following Wikipedias: Arabic, Czech and Hungarian. This deployment marks the first time the tool is available to people who have not enabled the feature in Beta Features or appended  to the URL of the talk page they are viewing.

The primary goal of this deployment is to understand whether contributors, across experience levels, are having success using the feature and finding it valuable. You can find more details about how we think about the deployment process here: Deployment process.

You can see the kinds of edits people at the Arabic, Czech and Hungarian Wikipedias are making with the Reply Tool by filtering Recent Changes:


 * Arabic: https://w.wiki/dgV
 * Czech: https://w.wiki/dT7
 * Hungarian: https://w.wiki/dT6

14 September 2020
Custom edit summaries

By the end of Thursday, 17-September-2020, anyone using the Reply Tool on a production wiki will be able to customize the edit summary that accompanies the comments they post with the Reply Tool.

This functionality can be accessed by clicking the  link that will appear beneath the text input area (see the "Custom edit summary implementation" screenshot).

28 August 2020
Custom edit summaries

Inspired by the feedback volunteers have shared [i][ii][iii][iv], the Reply Tool will soon offer people the ability to customize the edit summaries that accompany comments posted with it.

As part of implementing the custom edit summary functionality, we created a technical prototype*.

You can try the prototype by visiting this page: https://patchdemo.wmflabs.org/wikis/23fd7e0b373b74aceaf8ddec1d82ab09/w/index.php/Talk:Main_Page.

If there are comments and/or questions that the prototype brings to mind, we would value you sharing them on the talk page here: Topic:Vstemnhdi8w8iw95.

''*Note: the prototype linked above implements – what we currently understand to be – the core custom edit summary functionality. It intentionally is not opinionated about interface polish/presentation.''

7 August 2020
Deployment


 * On 6-August-2020, the Reply Tool became available as a Beta Feature at the following Wikipedias: Catalan, Chinese, Czech, Georgian, Korean, Serbian, Sorani Kurdish and Swedish.

31 July 2020
Metrics *You can read more detail about this analysis in the Metrics section below.
 * Earlier this week, the team completed an analysis that looked at how people on our four partner wikis have used the Reply tool between 31-March-2020 and 30-June-2020.
 * The key question we were trying to answer with this analysis was: Do the people who have tried the tool find it valuable?
 * In summary, we are confident that people* who tried the Reply tool do find the tool valuable.

22 July 2020

 * Metrics: the team is analyzing how people on our partner wikis have used the Reply Tool between 31-March-2020 and 30-June-2020. You can expect findings to be posted in the Metrics section below in the next two weeks.
 * Refinements: in response to the usability tests we ran in June, the team is working on a series of refinements/enhancements to the tool. These are listed below along with the conversations where we are discussing these enhancements:
 * Support for custom edit summaries: Topic:Vju7lfcav875rt8r.
 * Adjustments to the toolbar: Topic:Vq9yz2a4xa2upy4p

26 June 2020
Version 2.0 usability test findings


 * To evaluate whether the new features introduced in version 2.0 of the Replying tool are intuitive to Junior and Senior Contributors, we ran two usability tests that asked test participants to complete a series of tasks.
 * These test findings are now available for you to review: Version 2.0 prototype (usertesting.com + mediawiki.org).

17 June 2020
Version 2.0 deployment


 * Today, 17-June-2020, Version 2.0 of the Reply tool became available in the following places:
 * Version 2.0 is now available as a Beta Feature on our partner Wikipedias (Arabic, Dutch, French and Hungarian).
 * Version 2.0 is now available for testing on all wikis by adding the following parameter to any talk page URL:.

5 June 2020
Version 2.0 testing and deployment

Version 1.0 deployment
 * Usability tests being run on mw.org, at usertesting.com and on our four partner Wikipedias (Arabic, Dutch, French and Hungarian) are ongoing.
 * Once these tests are complete, findings will be summarized and posted to the Usability testing section below.


 * Version 1.0 of the tool is now available for testing as a Beta Feature on mediawiki.org. To turn it on, visit Beta Features in your preferences and make sure the box next to "Discussion tools" is checked (☑️).

26 May 2020
Version 2.0 testing

Version 2.0 deployment
 * Version 2.0 of the Reply tool is ready for testing. It introduces functionality for pinging other people by typing @ and writing and styling comments in a new  mode.
 * You can find instructions for how to try the new version of the Reply tool here: Version 2.0 test instructions.


 * As of today, Version 2.0 of the Reply tool is available to everyone at our partner Wikipedias (Arabic, Dutch, French and Hungarian) via a custom URL.
 * To try Version 2.0 of tool at these wikis, add the following text to any URL where the Reply tool is already available:.
 * Once we have evidence this new version is working as expected, it will become available as a Beta Feature on our partner wikis.

1 May 2020
Version 1.0 deployment


 * For ~4 weeks, Version 1.0 of the new Replying tool has been available as a Beta Feature on our partner wikis.
 * In line with Step 2 of our deployment process, we have been focused on making sure the Reply tool is available on pages where discussions happen outside of talk namespaces. To this end...
 * You can now use the Reply tool on any page where the  syntax is present.
 * This means people can now use the Reply tool on pages like Dutch Wikipedia's The Pub. You can see the comments people are posting there by reviewing these Recent Changes.
 * Within the next week, the Reply tool will be available on in all namespaces where the Signature button is present.
 * This will mean you can use the Reply tool on pages without needing the "New section" tab to be available as well. This will be especially helpful on pages like French Wikipedia's Le Bistro, where conversations are transcluded from other pages.
 * You can now use the Reply tool to reply to comments transcluded from another page.
 * This means it is now possible to use the Reply tool on pages like the Dutch Wikipedia's page for discussing article deletions (Wikipedia:Te_beoordelen pagina).

Version 2.0 development

Our two areas of focus right now related to Version 2.0 are:


 * 1) Implementing an easy way to @-mention/ping other people.
 * 2) Making it possible to switch back and forth between the   and   text input modes.

Thank you to Pbsouthwood, Pelagic, Samat and TheDJ for the efforts you have made offering design feedback about the version 2.0 mockups.

3 April 2020
Version 2.0 designs


 * We have designs ready for Version 2.0 of the new Replying tool that we would value your feedback on.
 * If you have any thoughts or questions, please post them on the talk page here: Design feedback: version 2.0 mockups.

31 March 2020
Version 1.0 deployment


 * Today, Version 1.0 of the new Replying tool became available as a Beta Feature on the following partner wikis: Arabic, Dutch, French and Hungarian.
 * To ensure the new tool works in ways people expect, we will work together with volunteers at these four Wikipedias over the next couple of weeks to monitor how the feature is being used before deciding deploying it to more people on these as an opt-out user Preference.
 * You can learn more about our deployment process below: Deployment process.

26 March 2020
Version 1.0 deployment


 * Version 1.0 of the Replying feature is currently being tested. Assuming the QA process does not surface any critical issues, the new replying tool will be available as a Beta Feature on the Arabic, Dutch, French and Hungarian Wikipedias as soon as next week.

13 March 2020
Version 1.0 deployment


 * The deployment of the the new replying workflow as a Beta Feature on Arabic, Dutch, French and Hungarian Wikipedias will be delayed for an unspecified amount of time. The team will share an update on this page as soon as a new date is set.

12 March 2020
Active

Version 1.0 deployment
 * On Monday, 16-March-2020, the team is planning to deploy Version 1.0 of the new workflow for replying to specific comments on talk pages as a Beta Feature on the following Wikipedias: Arabic, Dutch, French and Hungarian.
 * You can read more about how this deployment decision was made below: Deploying Version 1.0 as a Beta Feature to partner wikis.
 * If there are issues you think should be fixed that have not yet been prioritized, please mention something on the talk page: Talk:Talk pages project/replying. Note: the team is especially interested in links to examples where using the Replying tool disrupts content elsewhere on the talk page. This Phabricator task is a good example of the kind of "disruption" we have in mind: T246481.

Version 2.0 development
 * The team continues to develop the new functionality that will be introduced in Version 2.0. Functionality like drafting comments using a rich text/visual editing mode.

Recently completed


 * Version 1.0 instrumentation: The team recently completed implementing and testing the instrumentation that will help them evaluate whether contributors are having success using the new Replying feature and whether contributors are using the tool in ways that could be negatively impacting the experience of others.

28 February 2020
Active


 * Version 1.0 deployment: The team is working to resolve the highest priority issues that surfaced during the initial deployment. Once this is complete, the feature will be deployed as a Beta Feature to four target Wikipedias: Arabic, Dutch, French, and Hungarian. Before this happens, contributors to these wikis will be notified.
 * Version 2.0 designs: The team is working on the designs for version 2.0. You can see an early iteration of the designs here: Talk pages project/replying.
 * Note: the team will formally ask for feedback on these designs in the coming weeks by starting a new discussion thread on the talk page. Although, if you have feedback to share before then, the team would value you sharing it on the talk page.

Recently completed


 * Version 1.0 deployment: on Tuesday, 18-February, Version 1.0 of the feature became live via a custom URL for testing in production, on four Wikipedias: Arabic, French, Dutch and Hungarian.
 * We, along with many of you (thank you!), have been testing the feature extensively to make sure it appears where it should and behaves how we expect. You can read and add to the conversations where people are sharing feedback in these places: MediaWiki, Arabic Wikipedia, Dutch Wikipedia, French Wikipedia and Hungarian Wikipedia.
 * Deployment status and process: the team published information about why, how and where the Replying feature is and will be deployed here: Talk pages project/replying.

17 February 2020
Active


 * Version 1.0 deployment: this week, the team will deploy version 1.0 of the new replying workflow to the first set of production Wikipedias (Arabic, Dutch, French and Hungarian). For the time being, the new feature will only be accessible using a custom URL. This way, the team can safely test the feature without impacting anybody else's experience. If this testing goes well, the feature will then be deployed as an opt-in Beta Feature.
 * Version 2.0: the team is working on the design of the next iteration of the feature. If you would like to review and add your feedback on the latest design, please review this Phabricator task: T235593.

Recently completed


 * Finalized the list of features that will be included in the next iteration of the feature. You can review the list of features that are planned here:.

24 January 2020
Active


 * Finishing development of version 1.0 ahead of the team's planned deployment of the new replying workflow to four wikis, beginning as an opt-in Beta Feature, in mid- to late-February.
 * Adding instrumentation to the new replying workflow. This will enable the team to measure how contributors are engaging with the workflow once it is deployed and determine whether adjustments need to be made to make it more intuitive.

Recently completed


 * Finalized the user interface for version 1.0 of the feature, including how real time previewing will look and function. You can see a mockup for what it will look like once version 1.0 of the feature is deployed here: File:New replying mockup (v1.0).

17 December 2019
Active


 * Prototype: the team is running an on-wiki user test of version 1.0 of the prototype. We would value you trying it out! To try the prototype and participate in the test, please visit: Prototype version 1.0 usability test
 * Measurement: the team is drafting a plan for how we intend to measure the quantitative impact of the improved workflow for replying to specific comments.

Recently completed


 * User testing: the team recently completed two user tests. Both tests involved people familiar with Wikipedia and who have little or no experience participating on Wikipedia talk pages. The first test was meant to identify the challenges people face when trying to participate on talk pages using the existing experience. The second test is meant to identify usability issues with version 1.0 of the prototype for the improved workflow for replying to specific comments. Read more about these tests below: User testing.

Open questions
This section contains questions the team would value your input on.

Talk pages to test
Are there particularly long or complex talk pages you think the team should consider testing the Replying feature on?

If so, please share a link to the page(s) you are thinking of on the talk page here: Talk pages to test.

These links will be helpful in increasing the likelihood the Replying will work on your wiki once it is deployed there.

Objectives
This work is intended to make participating productively on talk pages easier and more intuitive for contributors.

"Easier" means more experienced contributors can participate in existing discussions with less effort, and "intuitive" means newer contributors do not need specialized knowledge to figure out how to add their thoughts to a conversation. Said in a different way: as a result of this work, both newer and more experienced contributors should report the workflow for participating in existing discussions to be "obvious" and "approachable."

It is important to note the mention of "participating productively" in the first paragraph of this section. We appreciate it is not enough to build tools that simply encourage contributors to "talk more"; they need to encourage people to work together to help improve the project they are discussing.

While we are still defining how to measure the impact of this feature, a key part of this work will involve figuring out how to understand the impact this new affordance has had on the quality of discussions on the talk pages it is deployed to.

Metrics
''This section discusses the three quantitative analyses we have planned to evaluate the usefulness, usability and impact of the Reply Tool. These analyses are discussed in detail below.''

Analysis 1: Adoption

To start, we will seek to learn whether people at the Arabic, Dutch, French and Hungarian Wikipedias, where the Reply Tool has been available as a Beta Feature since 31-March-2020, are finding the tool valuable.

To determine the extent to which people are "finding the tool valuable" we will look at how frequently people are using the tool (as measured by the number of distinct days they use it on) and how intensely people are using the tool (as measured by the total number of edits they make with the Reply Tool and the percentage of total talk page edits they used the Reply Tool to make).

Understanding the above will help us determine whether the tool is ready to be made available as an opt-out feature at these wikis and made available as an opt-in Beta Feature at others.

Timing

This analysis was completed on 28-July-2020.

Findings

The data below includes people who have used the Reply Tool as a Beta Feature at the Arabic, Dutch, French and Hungarian Wikipedias, between 31-March and 30-June-2020: Report
 * 328 people posted at least one comment using the Reply Tool
 * 69.5% of people posted ≥ 2 comments with the Reply Tool
 * 61.3% of people posted a comment using the Reply Tool on ≥ 2 distinct days*
 * 23.6% of people used the Reply Tool to make over half of their talk page edits**

You can review the full analysis results in this Jupyter notebook: https://nbviewer.jupyter.org/github/wikimedia-research/Discussion-tools-analysis-2020/blob/master/Replying-Tool-Adoption-Metrics.ipynb.

'' * A "distinct day" was defined as a distinct calendar day. Therefore, some of these edits may have occurred 24 hours apart and some may have occurred only a few hours apart depending on what time of day the edit was made.''

**This data is a bit noisy in so far as there are some edits that cannot be made with the Reply Tool (e.g. starting a new section, editing existing content, etc.) and the software is not currently able to differentiate between comments and other talk page edits.

Analysis 2: Engagement

The next analysis we do will be intended to help determine whether people, across experience levels, are having success using the Reply Tool and whether people are using the tool in ways that degrades the experiences of others.

This analysis's findings will be used to decide whether the Reply Tool is functioning well enough for its impact on user behavior to be tested via a larger-scale (read: at wikis beyond our partner wikis) A/B test.

Timing

This analysis will happen after the Reply Tool has been made available as an opt-out feature at our partner wikis.

Analysis 3: Impact

The last analysis we have planned will be used to determine whether the Reply Tool causes a greater percentage of Junior Contributors to publish the comments they start without causing a significant increase in disruption to other people.

This analysis will be used to help us decide whether the Reply Tool can be made available to all people, at all projects, by default. More details can be found in Phabricator here: T252057.

Timing

This A/B test is scheduled to begin in January, 2021 at 22 Wikipedias. The list of wikis can be found in Phabricator here: T267382.

Background
This year, the Editing Team is committed to improving how contributors communicate and collaborate on Wikipedia, using talk pages.

For talk pages to be valuable, contributors need to intuitively know how to participate in the conversations that happen on them.

The trouble is – as previous user testing, the Talk Page Consultation and the team's research uncovered – contributors, across experience levels, find replying to specific comments on Talk pages to be challenging.

Specifically, the team's research has found:


 * More experienced contributors find it difficult to locate the specific comment they are wanting to reply to when participating in long, multi-person conversations, within big blocks of wikitext.
 * Newer contributors report not being sure how to reply to a conversation, regardless of its length or complexity.

In exploring an affordance for replying to specific comments on talk pages, we are striving to make participating productively on talk pages easier and more intuitive. We think doing so will help newer contributors understand and use talk pages as places to communicate with others and help more senior contributors collaborate more efficiently.

Challenges
One part of building new features is codifying social conventions in software. In this context, "conventions" could mean deciding what character the software should use for automatically indenting or outdenting replies. "Conventions" could also mean deciding how the software should represent the first comment in a discussion in wikitext: ''Is there a linebreak between the reply and the original comment? Is the reply added to the line immediately following the original comment in the discussion? Is there another way this comment should be represented?''

It is unlikely all communities will answer these questions in the same way. This means the software needs to be built in a flexible enough way to accommodate these different preferences. This is possible, although it adds complexity to the design and development processes.

Design
To increase the likelihood this enhancement is useful for contributors across experience levels, we have broken down the improvements we have planned into a series of releases. These different releases are outlined in more detail in the "Versions" section below.

Also below are the latest iteration for the designs we are planning to implement in Version 1.0. If you have thoughts about anything included in this section, we would value you sharing them on the talk page: Talk:Talk pages project/replying.

❗️Please consider the features included below as drafts and expect them to evolve as we learn new things.

Version 1.0
This version will introduce the basic reply functionality to validate the core workflow. This version will likely include the following features:


 * An affordance for replying to specific comments
 * A text box for composing replies using wikitext
 * A way for signatures to be appended to comments, automatically
 * A way for those comments to be indented or outdented, automatically

You can try a prototype of version 1.0 of the new replying workflow here: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Cats.

A mockup showing how version 1.0 will likely look once it is deployed can be seen in this screenshot: File:New replying mockup (v1.0).png

Version 2.0
Assuming version 1.0, and any smaller releases that follow, helps us to validate and refine the core replying functionality, version 2.0 will likely include enhancements to make it easier and intuitive for Junior Contributors to draft and post their replies. This version will likely include the addition of the following features:


 * Opt-in/out of watching the talk page as part of the replying workflow
 * Populating the reply text input with instructional text to make it clear who you are responding to.
 * Drafting and formatting replies using a rich text editor
 * Switching between "rich text" and "source" writing modes while preserving any content contributors have already written
 * Mentioning/notifying specific users in a conversation without needing to know about or interact with wikitext
 * Editing a specific comment
 * Note: It is not likely editing a specific comment will be included in the initial release of version 2.0. This is due to the technical complexity involved with implementing this functionality. You can learn more about this "technical complexity" and the initial implementation we have planned in Phabricator here: T245225.

Technical information
The replying feature is implemented via the DiscussionTools extension.

Troubleshooting information is available at Help:DiscussionTools/Why can't I reply to this comment?

Deployment
This section contains information about how and where the Replying feature is and will be deployed.

Deployment decisions
Deploying Version 1.0 as a Beta Feature to partner wikis

On Monday, 16-March-2020, the team is planning to deploy Version 1.0 of the new workflow for replying to specific comments on talk pages as a Beta Feature on the following Wikipedias: Arabic, Dutch, French and Hungarian.

This decision is the outcome of the team being confident in the following:


 * Reply links are available in the places (read: talk pages, across namespaces) where people expect them to be.
 * The core parts of the Replying workflow (e.g. writing a comment, previewing a comment, formatting a comment, discarding a comment and publishing a comment) are working in the ways people expect.
 * The Replying workflow is not disrupting existing talk page content.

This confidence is based on the team and volunteers testing the new Replying workflow on >75 different talk pages in three talk namespaces on the 4 Wikipedias where the feature will be deployed.

This testing proved necessary for identifying and resolving issues related to the core parts of the replying workflow. Issues like:


 * The replying text box remaining after cancelling a reply: T245574.
 * Reply links appearing after list items that were not comments: T245692 (thank you, User:Bdijkstra).
 * Reply links not appearing after comments that contained signatures with an invisible Unicode character: T245784.
 * The reply tool not working in Safari: T245781.
 * The reply tool removing a page from your watchlist: T245579 (thank you, User:Tacsipasci).
 * Broken and incomplete table syntax causing page corruption: T246481.

This testing was also important for identifying additional use cases the team is actively working to provide support for. Use cases like being able to reply to comments transcluded from a subpage. For more details on this work, see this Phabricator task: T245694.

Deployment process
The deployment of this feature will happen in four steps. These four steps are described below.

Before the team decides to move on to the next step in the deployment process, they will do two things:


 * 1) Make contributors affected by the next planned deployment aware of where to test the version of the feature the team is planning to deploy, how the feature will be deployed and what informed the deployment decision. Announcements containing this information will be posted as updates on this project page and publicly on all wikis affected by the planned deployment.
 * 2) Invite contributors to share their input about the next deployment the team is planning. Contributors will be invited to share their input about the next deployment on the talk page and on pages where the deployment is announced.

Step 1: Beta cluster
Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Do contributors find replying to comments using the feature intuitive?
 * Are any parts of the workflow broken (e.g. writing and formatting a reply, cancelling a reply, previewing a reply, publishing a reply)?

Deployment details

To make sure the feature behaves in ways contributors expect, new versions of the feature will be deployed to the Beta Cluster. This way, contributors will be able to safely experiment with new versions of the Replying feature without needing to worry about affecting existing content or contributors.

Step 2: Beta Feature
Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Is the feature available where it should be?
 * Does the feature affect existing talk page content in unintended ways?
 * Are there any issues that prevent contributors from being able to respond to comments?

Deployment details

To make sure the feature behaves in ways contributor expect on production wikis before exposing the feature to a large number of people, the feature will be deployed as a Beta Feature.

Step 3: User Preference (opt-out)
Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Are people able to successfully post replies using the feature?
 * Is the feature having any unintended affects (e.g. encouraging disruptive talk page comments edits)?
 * Are people finding the feature valuable?

Deployment details

To determine whether a broad range of people are having success using the feature and are finding it valuable, the feature will be deployed as an opt-out User Preference.

Step 4: A/B test
Deployment goal

The goal of this step in the deployment process is to understand whether the feature had the impact it was intended to have. To determine this, we will use this deployment step to answer:


 * Does the new replying feature make it easier and more intuitive for Junior and Senior Contributors to respond to specific comments on Wikipedia talk pages?

Deployment details

To determine whether the feature had the impact it was intended to have, the team will run an A/B test on a to-be-determined set of wikis.

The team is finalizing the details of this A/B test.

Step 5: All wikis
If evidence from the previous 4 deployment steps suggests the new Replying feature makes replying to specific comments on talk pages easier and more intuitive for Junior and Senior Contributors, it will be deployed to all remaining Wikipedias. Determining how and when the feature is deployed depends on the team, along with volunteers from those wikis, being confident the feature works reliably and in ways people expect it to.

Usability testing
This section contains information about user testing the team conducts to ensure the revised experience works in ways contributors expect.

Version 2.0 prototype
To evaluate whether the new features introduced in version 2.0 of the Replying tool are intuitive to Junior and Senior Contributors, we ran two usability tests that asked test participants to complete a series of tasks:


 * One test recruited Junior Contributors (read: people familiar with reading Wikipedia and who have little-to-no experience using Wikipedia talk pages) via usertesting.com.
 * One test recruited Senior Contributors (read: people who have experience using Wikipedia talk pages) via mediawiki.org.

Overall, these tests demonstrated the majority of Junior and Senior Contributors were successful in using Version 2.0 of the Reply tool to write, format and publish comments on Wikipedia talk pages.

You can learn about the overall impressions people had, the key areas for improvement and next steps below.

Overal impressions

Senior Contributors described the tool as well-balanced and expressed appreciation for the new automatic pinging feature. Here are a few noteworthy comments:


 * "I can insert a link to an [sic] user page inserting an @, after which a list of usernames pops up. That is nice."
 * "...this [pinging] is a game changer."
 * "Nice tool for both beginners and experienced users."
 * "Good tool, simple to use, it's good for what it's made for."

Junior Contributors who were able to find the Reply tool successfully completed the tasks they were asked to and described the tool as being intuitive and straightforward to use.

With this said, when peoples' focus moved from the Reply tool to the broader talk page, they became confused and uncertain about "where" they were and what they ought to do.* Here are a few comments we thought were noteworthy:


 * "It was pretty straightforward. streamlined process here i like the fact that it's very stripped down and doesn't feel data intense.so not overwhelming by any stretch of the imagination. the colors are pleasing to the eye... simple to navigate interface. Overall it works the way i'd expect it to."
 * "I'm a little confused about what User talk: Alice is..."
 * "Doesn't look like the sort of usual messaging interface you might see on a website."

* Note: Junior Contributors not recognizing talk pages as places to communicate with other editors is a finding that continues to surface[1] and something we plan to address later in the Talk pages project.

Task analysis

The table below shows whether participants in each test group were able to complete the following tasks with Version 2.0 of the Reply tool.*

You can review the broader themes these tests helped our team identify in the "Key themes" section below.

*Note these test findings were compiled on 10-June-2020.

Areas for improvement

These are the broader areas for improvement that surfaced through these usability tests. You can review how we plan to address these themes in the "Next steps" section below.


 * 1) The tool's two modes:   and
 * 2) * Some Senior Contributors were surprised to learn when viewing a diff that the  mode automatically signed the comment they posted.
 * 3) * Some Junior Contributors did not find the difference between the  and   modes to be clear.
 * 4) Pinging
 * 5) * Some Senior Contributors said they would not have known the pinging feature was available had they not been instructed to try it.
 * 6) * Some Senior Contributors wondered whether Junior Contributors would be confused by the normal link inspector that is presented when a ping within a drafted comment is selected.
 * 7) Commenting in long threads
 * 8) * Senior Contributors found it can be easy to "lose the tool" when trying to comment in a conversation and/or on a page with many comments.
 * 9) ** An example: when trying to respond to a comment that has many existing replies, it can be difficult to simultaneously look at the comment you are responding to and the comment you are drafting using the Reply tool.
 * 10) * Senior Contributors found the tools within the  mode hard to "reach" when working on a large screen and when writing a lengthier comment.
 * 11) Edit summaries
 * 12) * Some Senior Contributors wondered whether supports for custom edit summaries would be added.

Next steps

To address the issues and opportunities these two usability tests surfaced, the team will be working on improvements to make the following come true;


 * Contributors, across experience levels, can easily mention/ping another person in the comment they are writing.
 * Example improvement: adding an icon to the  mode's toolbar that communicates there is a feature for pinging people.
 * Contributors, across experience levels, know how the tool's two input modes function and how they relate to one another.
 * Example improvement: revising what the  /  modes are called and how prominently they appear in the interface.
 * Contributors, across experience levels, find the tool well suited for commenting in talk page conversations, regardless of how many conversations (read: sections) and comments they contain.
 * Example improvement: revising where the text input area appears on the page and how it behaves when the page is scrolled.

You can track the progress we make on the areas above by watching this page and/or by reading this Phabricator workboard: phab:Reply tool version 2.0.

Version 1.0 prototype
To see how the version 1.0 prototype affected Junior Contributors' experience replying in conversations on talk pages, the team ran a control test with 5 participants on usertesting.com. You can review the test findings below.

How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia before.

In order to compare the revised replying workflow to the [https://www.mediawiki.org/wiki/Talk_pages_project/replying#What_did_we_find? existing workflow], each test participant was asked to complete the same tasks on a desktop computer, while narrating their experience:


 * 1) Navigate to an existing discussion happening on a test wiki talk page
 * 2) Draft and publish a response in that discussion
 * 3) Locate their published response on the talk page
 * 4) Navigate to the talk page's history page and locate the reply they had just posted
 * 5) Start drafting another reply, but discard it before publishing

What did we find?
Overall, the prototype seems to have improved Junior/newer contributors' experiences replying in an existing conversations. On average, it took participants using the prototype half the time to publish a reply compared to the time it took them in the previous test, using full page editing.

Task completion


 * ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
 * ✅All participants successfully composed and published a reply in an existing discussion
 * ✅All participants properly indented and signed their comments
 * Note: the tool does this for them automatically.
 * ✅All participants were able to locate their published response on the talk page
 * ✅All participants were able to find the talk page history page
 * ❌1 out of 5 participants was able to quickly locate the edit they had just made on the talk page's history page
 * ✅All participants were able to cancel their second reply before publishing it

Results

The prototype seems to have made it easier for Junior/newer contributors to reply to existing conversations on talk pages. Test participants used phrases like, "straight-forward," "no-problem whatsoever," and "really easy" to describe their experiences.

With this said, there are still parts of the replying workflow participants found difficult that could be improved:


 * Making it easier to visually distinguish between different replies in a discussion
 * Making it easier to identify who the author of a comment is and who they are responding to
 * Making the "Reply" call to action easier to discover
 * Revising the automatically generated edit summary to make it easier for contributors to identify their reply on the History page.

More details can be found in this ticket on Phabricator: T236921#5744471.

Next steps

The team will decide if and when to make the improvements mentioned above (e.g. making it easier to visually distinguish between different replies in a discussion) and do further testing with more experienced contributors (this has started here: Version 1.0 prototype test).

Existing reply experience
To identify the challenges Junior Contributors face when trying to participate in conversations on talk pages, the team ran a control test of the existing (full page) editing workflow.

How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia in some capacity before.

Each test participant was asked to do the following tasks on a desktop computer, while narrating their experience:


 * 1) Navigate to an existing discussion happening on a test wiki talk page
 * 2) Draft and publish a response in that discussion
 * 3) Locate their published response on the talk page
 * 4) Start drafting another reply, but discard it before publishing
 * 5) Navigate to the talk page's history page and locate the reply they had just posted

What did we find?
Below is a summary of our findings from this user test. More details can be found in this ticket on Phabricator: T239175#5723843.

Task completion


 * ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
 * ✅All participants successfully composed and published a reply in an existing discussion
 * ❌0 out of 5 participants properly indented and signed their comments
 * ❌1 out of 5 participants noticed they had not signed their comments
 * This one person remarked, "I probably should have put my user name, but i didn't".
 * 3 out of 5 participants were able to locate their published response on the talk page.
 * Note: it took several participants multiple minutes to do this
 * ✅4 out of 5 participants were able to find the talk page history page and locate the reply they had just posted
 * ✅4 out of 5 participants were able to cancel their second reply before publishing it

Results

This test highlighted an important tension many Junior Contributors seem to face: they finish the task they set out to complete without realizing they might have done so incorrectly. And if they do realize they have made a mistake, they are not equipped to fix it because the proper conventions are not intuitive enough for them to understand.

Next steps

The team will be doing two things in response to this test:


 * The team will be doing additional testing in different namespaces and languages (see: Partner wikis).
 * We will run a usability test of the prototype with contributors, across experience levels, to see how it affects their experiences replying to comments on talk pages.

History
Many projects have, and are, working to improve contributors' experiences with talk pages. This project is better off for their existence. Some of the projects the team continues to learn from are listed on the main project page and below. If there is a project you think we should be aware of, please boldly add it here.


 * Convenient Discussions
 * Flow satisfaction survey
 * Moderated Testing, November 2014: talk pages and Flow
 * Reply-link
 * Talk pages consultation

Glossary
The Talk pages project glossary is intended to help us all communicate about talk pages more effectively by making sure we have a shared understanding about the words we use in our discussions and documentation throughout the project.