Talk pages project/Usability/Analysis

This page outlines the Editing Team's proposal for how we – volunteers and staff – will assess the impact that changes to how desktop talk pages appear have on volunteers across experiences levels.

Developing a shared understanding for how we will evaluate this impact is crucial for deciding whether these changes are fit for being made available to everyone who uses article  and   pages.

Objectives
The changes to how talk pages appear have been designed to cause:


 * 1) Junior Contributors to quickly recognize talk pages as places to communicate with other volunteers and locate the tools available to do so.
 * 2) Senior Contributors to be able to quickly assess which conversations on a given talk page are worth focusing on.

Evaluation criteria
To evaluate the impact of the set of changes to how talk pages appear, and the information about the discussions contained within them, the Editing Team is interested in two categories of metrics:


 * Key performance indicators will be used to determine the success of the features and ultimately, decide whether we think the Usability Improvements, in their current state, warrant being made available by default.
 * 1) Curiosities will be used to learn what – if any – other effects the Usability Improvements caused. This information will not be used to determine the success of the features. Rather, this information will be used to inform our broader understanding of these interventions and potential opportunities for further improvement in the future.

See T302358 for more information about the metrics above.

Experiment plan
To evaluate the collective impact of the set of changes that comprise the Talk pages project's Usability Improvements, the Editing Team is planning to do the following.

Feature overview
Usability Improvements is made up of four distinct changes to how talk pages appear.


 * 1) Headings appear in a sans serif font and gain three pieces of metadata about the discussions contained within them:
 * 2) the date the last comment was published in the discussion,
 * 3) the number of comments within the discussion, and
 * 4) the total number of unique people who have published a comment in the discussion.
 * 5) Table of contents includes information about the total number of comments within each discussion on the page.  (This change only affects Vector 2022.)
 * 6) Buttons for replying and starting a new topic appear in a bolder type face.
 * 7) Breadcrumb is added to the top of talk pages that includes when the last comment was published on the page, the user name of the person who published that comment, and the discussion that comment was published within.

The four changes described above were designed together to have the impact described in the Objectives section above.

Motivation
For wikitext talk pages to be valuable, Junior Contributors need to intuitively recognize them as places to communicate with other volunteers, and Senior Contributors need to be able to spend minimal effort understanding the conversations happening on a given page.

The trouble is, volunteers across experience levels report that the current presentation of wikitext talk pages can make recognizing talk pages, understanding the conversations happening within them, and identifying what they need to click/tap to participate in these conversations unnecessarily difficult.

Specifically, user feedback,      usability tests,  the Talk Page Consultation (see: #Newcomers, #Confusion, and #Design), and academic research have highlighted the difficulties people have reading and using talk pages.

By making the components that comprise talk page conversations (e.g., conversation topics, conversation boundaries, comments, etc.) and the tools necessary for participating in them easier for people to parse, we are striving to make it easier for volunteers, across experience levels, to communicate on-wiki.

More information about the research that motivated this work can be found on the project page.