Talk pages project/Replying

This page talks about the Editing Team's work to improve contributors' workflows for replying to comments on talk pages, across Wikipedia's 16 talk namespaces. This new workflow for replying to specific comments is intended to make participating productively on talk pages easier and more intuitive for Junior and Senior Contributors.This initiative sits within our team's larger effort to help contributors work together more effectively. To accomplish this, we will build upon existing community conventions to evolve talk pages. It is our intention to evolve talk pages in a way that gives experienced contributors more leverage to coordinate their work and connect with other editors, while making communicating on-wiki more accessible and intuitive for newer contributors.

To participate in and follow this project's development, we recommend adding this page to your watchlist. We will use this page to do things like:


 * Share potential designs and ask for feedback about them
 * Test prototypes
 * Post updates about this feature's development
 * Ask and invite questions about any part of this project (e.g., measurement, deployments, etc.)

Status updates
This section contains updates about the project's development.

17 February 2020
Active


 * Version 1.0 deployment: this week, the team will deploy version 1.0 of the new replying workflow to the first set of production Wikipedias (Arabic, Dutch, French and Hungarian). For the time being, the new feature will only be accessible using a custom URL. This way, the team can safely test the feature without impacting anybody else's experience. If this testing goes well, the feature will then be deployed as an opt-in Beta Feature.
 * Version 2.0: the team is working on the design of the next iteration of the feature. If you would like to review and add your feedback on the latest design, please review this Phabricator task: T235593.

Recently completed


 * Finalized the list of features that will be included in the next iteration of the feature. You can review the list of features that are planned here:.

24 January 2020
Active


 * Finishing development of version 1.0 ahead of the team's planned deployment of the new replying workflow to four wikis, beginning as an opt-in Beta Feature, in mid- to late-February.
 * Adding instrumentation to the new replying workflow. This will enable the team to measure how contributors are engaging with the workflow once it is deployed and determine whether adjustments need to be made to make it more intuitive.

Recently completed


 * Finalized the user interface for version 1.0 of the feature, including how real time previewing will look and function. You can see a mockup for what it will look like once version 1.0 of the feature is deployed here: File:New replying mockup (v1.0).

17 December 2019
Active


 * Prototype: the team is running an on-wiki user test of version 1.0 of the prototype. We would value you trying it out! To try the prototype and participate in the test, please visit: Prototype version 1.0 usability test
 * Measurement: the team is drafting a plan for how we intend to measure the quantitative impact of the improved workflow for replying to specific comments.

Recently completed


 * User testing: the team recently completed two user tests. Both tests involved people familiar with Wikipedia and who have little or no experience participating on Wikipedia talk pages. The first test was meant to identify the challenges people face when trying to participate on talk pages using the existing experience. The second test is meant to identify usability issues with version 1.0 of the prototype for the improved workflow for replying to specific comments. Read more about these tests below: User testing.

Open questions
This section contains questions the team would value your input on. If you have thoughts to share, please add them to the talk page here: Talk:Talk pages project/replying

Question #1: Previews
What do you find valuable about being able to preview the content of a reply you are drafting before posting it to a talk page? If you have thoughts, the team would value your thoughts here: Topic:Vcehezaiyl3znf0d


 * Context for this question: The team is thinking through how the initial workflow for "previewing" the reply you are composing might work as part of this new dedicated workflow for replying to specific comments. For reference, the way User:Thnidu articulates why they value previews has been instructive: "...a large part of their value for me is spotting errors of execution as well as errors of intention." [1]

Question #2: Indentations
''What wikitext should the reply tool use for the indentations it will automatically generate? Does the tool use whatever indentation syntax is used on the page the tool is currently being used on? Does it use  in all cases? Something else?'' If you have thoughts, the team would value your thoughts here: Topic:Vcehgf0nr8rpi26a


 * Context for this question: When bullets are used for indentations (e.g. on ru.wiki), multi-line comments will render on the page as if they are separate comments. This situation can be avoided if contributors manually insert  tags as has been done here. Assuming there will be contributors using the reply tool who will not know or think to manually insert    tags, what can be done to make sure their multi-line comments are posted to pages in a way that is legible to others?

Objectives
This work is intended to make participating productively on talk pages easier and more intuitive for contributors.

"Easier" means more experienced contributors can participate in existing discussions with less effort, and "intuitive" means newer contributors do not need specialized knowledge to figure out how to add their thoughts to a conversation. Said in a different way: as a result of this work, both newer and more experienced contributors should report the workflow for participating in existing discussions to be "obvious" and "approachable."

It is important to note the mention of "participating productively" in the first paragraph of this section. We appreciate it is not enough to build tools that simply encourage contributors to "talk more"; they need to encourage people to work together to help improve the project they are discussing.

While we are still defining how to measure the impact of this feature, a key part of this work will involve figuring out how to understand the impact this new affordance has had on the quality of discussions on the talk pages it is deployed to.

Metrics
This section will contain the metrics we will use to measure the impact of this feature.

Background
This year, the Editing Team is committed to improving how contributors communicate and collaborate on Wikipedia, using talk pages.

For talk pages to be valuable, contributors need to intuitively know how to participate in the conversations that happen on them.

The trouble is – as previous user testing, the Talk Page Consultation and the team's research uncovered – contributors, across experience levels, find replying to specific comments on Talk pages to be challenging.

Specifically, the team's research has found:


 * More experienced contributors find it difficult to locate the specific comment they are wanting to reply to when participating in long, multi-person conversations, within big blocks of wikitext.
 * Newer contributors report not being sure how to reply to a conversation, regardless of its length or complexity.

In exploring an affordance for replying to specific comments on talk pages, we are striving to make participating productively on talk pages easier and more intuitive. We think doing so will help newer contributors understand and use talk pages as places to communicate with others and help more senior contributors collaborate more efficiently.

Challenges
One part of building new features is codifying social conventions in software. In this context, "conventions" could mean deciding what character the software should use for automatically indenting or outdenting replies. "Conventions" could also mean deciding how the software should represent the first comment in a discussion in wikitext: ''Is there a linebreak between the reply and the original comment? Is the reply added to the line immediately following the original comment in the discussion? Is there another way this comment should be represented?''

It is unlikely all communities will answer these questions in the same way. This means the software needs to be built in a flexible enough way to accommodate these different preferences. This is possible, although it adds complexity to the design and development processes.

Design
To increase the likelihood this enhancement is useful for contributors across experience levels, we have broken down the improvements we have planned into a series of releases. These different releases are outlined in more detail in the "Versions" section below.

Also below are the latest iteration for the designs we are planning to implement in Version 1.0. If you have thoughts about anything included in this section, we would value you sharing them on the talk page: Talk:Talk pages project/replying.

❗️Please consider the features included below as drafts and expect them to evolve as we learn new things.

Versions
Version 1.0 This version will introduce the basic reply functionality to validate the core workflow. This version will likely include the following features:


 * An affordance for replying to specific comments
 * A text box for composing replies using wikitext
 * A way for signatures to be appended to comments, automatically
 * A way for those comments to be indented or outdented, automatically

You can try a prototype of version 1.0 of the new replying workflow here: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Cats.

A mockup showing how version 1.0 will likely look once it is deployed can be seen in this screenshot: File:New replying mockup (v1.0).png

Version 2.0

Assuming version 1.0, and any smaller releases that follow, helps us to validate and refine the core replying functionality, version 2.0 will likely include enhancements to make it easier and intuitive for Junior Contributors to draft and post their replies. This version will likely include the addition of the following features:


 * Opt-in/out of watching the talk page as part of the replying workflow
 * Populating the reply text input with a link to the user page of the person you are responding to
 * Drafting and formatting replies using a rich text editor
 * Switching between "rich text" and "source" writing modes while preserving any content contributors have already written
 * Mentioning/notifying specific users in a conversation without needing to know about or interact with wikitext
 * Editing a specific comment

Deployment
This section contains information about how and where the Replying feature is and will be deployed.

Deployment process
The deployment of this feature will happen in four steps. These four steps are described below.

Before the team decides to move on to the next step in the deployment process, they will do two things:


 * 1) Make contributors affected by the next planned deployment aware of where to test the version of the feature the team is planning to deploy, how the feature will be deployed and what informed the deployment decision. Announcements containing this information will be posted as updates on this project page and publicly on all wikis affected by the planned deployment.
 * 2) Invite contributors to share their input about the next deployment the team is planning. Contributors will be invited to share their input about the next deployment on the talk page and on pages where the deployment is announced.

Step 1: Beta cluster
Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Do contributors find replying to comments using the feature intuitive?
 * Are any parts of the workflow broken (e.g. writing and formatting a reply, cancelling a reply, previewing a reply, publishing a reply)?

Deployment details

To make sure the feature behaves in ways contributors expect, new versions of the feature will be deployed to the Beta Cluster. This way, contributors will be able to safely experiment with new versions of the Replying feature without needing to worry about affecting existing content or contributors.

Step 2: Beta Feature
Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Is the feature available where it should be?
 * Does the feature affect existing talk page content in unintended ways?
 * Are there any issues that prevent contributors from being able to respond to comments?

Deployment details

To make sure the feature behaves in ways contributor expect on production wikis before exposing the feature to a large number of people, the feature will be deployed as a Beta Feature.

Step 3: User Preference (opt-out)

Deployment goal

The goal of this step in the deployment process is to answer these questions:


 * Are people able to successfully post replies using the feature?
 * Is the feature having any unintended affects (e.g. encouraging disruptive talk page comments edits)?
 * Are people finding the feature valuable?

Deployment details

To determine whether a broad range of people are having success using the feature and are finding it valuable, the feature will be deployed as an opt-in User Preference.

Step 4: A/B test
Deployment goal

The goal of this step in the deployment process is to understand whether the feature had the impact it was intended to have. To determine this, we will use this deployment step to answer:


 * Does the new replying feature make it easier and more intuitive for Junior and Senior Contributors to respond to specific comments on Wikipedia talk pages?

Deployment details

To determine whether the feature had the impact it was intended to have, the team will run an A/B test on a to-be-determined set of wikis.

The team is finalizing the details of this A/B test.

Step 5: All wikis
If evidence from the previous 4 deployment steps suggests the new Replying feature makes replying to specific comments on talk pages easier and more intuitive for Junior and Senior Contributors, it will be deployed to all remaining Wikipedias. Determining how and when the feature is deployed depends on the team, along with volunteers from those wikis, being confident the feature works reliably and in ways people expect it to.

User testing
This section contains information about user testing the team conducts to ensure the revised experience works in ways contributors expect and need it to.

Version 1.0 prototype (usertesting.com)
To see how the version 1.0 prototype affected Junior Contributors' experience replying in conversations on talk pages, the team ran a control test with 5 participants on usertesting.com. You can review the test findings below.

How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia before.

In order to compare the revised replying workflow to the [https://www.mediawiki.org/wiki/Talk_pages_project/replying#What_did_we_find? existing workflow], each test participant was asked to complete the same tasks on a desktop computer, while narrating their experience:


 * 1) Navigate to an existing discussion happening on a test wiki talk page
 * 2) Draft and publish a response in that discussion
 * 3) Locate their published response on the talk page
 * 4) Navigate to the talk page's history page and locate the reply they had just posted
 * 5) Start drafting another reply, but discard it before publishing

What did we find?
Overall, the prototype seems to have improved Junior/newer contributors' experiences replying in an existing conversations. On average, it took participants using the prototype half the time to publish a reply compared to the time it took them in the previous test, using full page editing.

Task completion


 * ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
 * ✅All participants successfully composed and published a reply in an existing discussion
 * ✅All participants properly indented and signed their comments
 * Note: the tool does this for them automatically.
 * ✅All participants were able to locate their published response on the talk page
 * ✅All participants were able to find the talk page history page
 * ❌1 out of 5 participants was able to quickly locate the edit they had just made on the talk page's history page
 * ✅All participants were able to cancel their second reply before publishing it

Results

The prototype seems to have made it easier for Junior/newer contributors to reply to existing conversations on talk pages. Test participants used phrases like, "straight-forward," "no-problem whatsoever," and "really easy" to describe their experiences.

With this said, there are still parts of the replying workflow participants found difficult that could be improved:


 * Making it easier to visually distinguish between different replies in a discussion
 * Making it easier to identify who the author of a comment is and who they are responding to
 * Making the "Reply" call to action easier to discover
 * Revising the automatically generated edit summary to make it easier for contributors to identify their reply on the History page.

More details can be found in this ticket on Phabricator: T236921#5744471.

Next steps

The team will decide if and when to make the improvements mentioned above (e.g. making it easier to visually distinguish between different replies in a discussion) and do further testing with more experienced contributors (this has started here: Version 1.0 prototype test).

Existing reply experience (usertesting.com)
To identify the challenges Junior Contributors face when trying to participate in conversations on talk pages, the team ran a control test of the existing (full page) editing workflow.

How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia in some capacity before.

Each test participant was asked to do the following tasks on a desktop computer, while narrating their experience:


 * 1) Navigate to an existing discussion happening on a test wiki talk page
 * 2) Draft and publish a response in that discussion
 * 3) Locate their published response on the talk page
 * 4) Start drafting another reply, but discard it before publishing
 * 5) Navigate to the talk page's history page and locate the reply they had just posted

What did we find?
Below is a summary of our findings from this user test. More details can be found in this ticket on Phabricator: T239175#5723843.

Task completion


 * ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
 * ✅All participants successfully composed and published a reply in an existing discussion
 * ❌0 out of 5 participants properly indented and signed their comments
 * ❌1 out of 5 participants noticed they had not signed their comments
 * This one person remarked, "I probably should have put my user name, but i didn't".
 * 3 out of 5 participants were able to locate their published response on the talk page.
 * Note: it took several participants multiple minutes to do this
 * ✅4 out of 5 participants were able to find the talk page history page and locate the reply they had just posted
 * ✅4 out of 5 participants were able to cancel their second reply before publishing it

Results

This test highlighted an important tension many Junior Contributors seem to face: they finish the task they set out to complete without realizing they might have done so incorrectly. And if they do realize they have made a mistake, they are not equipped to fix it because the proper conventions are not intuitive enough for them to understand.

Next steps

The team will be doing two things in response to this test:


 * The team will be doing additional testing in different namespaces and languages (see: Partner wikis).
 * We will run a usability test of the prototype with contributors, across experience levels, to see how it affects their experiences replying to comments on talk pages.

History
Many projects have, and are, working to improve contributors' experiences with talk pages. This project is better off for their existence. Some of the projects the team continues to learn from are listed on the main project page and below. If there is a project you think we should be aware of, please boldly add it here.


 * Convenient Discussions
 * Flow satisfaction survey
 * Moderated Testing, November 2014: talk pages and Flow
 * Reply-link
 * Talk pages consultation

Glossary
The Talk pages project glossary is intended to help us all communicate about talk pages more effectively by making sure we have a shared understanding about the words we use in our discussions and documentation throughout the project.