Talk pages project/Impact

From mediawiki.org

This page contains an overview of the impact of the new functionality the Editing Team introduced through the Talk Pages Project.

Overall Impact[edit]

A line graph showing the proportion of talk page edit people have made with DiscussionTools over time.
Proportion of talk page edits made DiscussionTools at ar.wiki over time. Explore usage across wikis here: https://dtstats.toolforge.org.

Mobile[edit]

Analysis 1: Reply and New Topic Tools Impact[edit]

The rates at which people who are logged in and out publish the mobile talk page edits they start before and after the mobile Reply and New Topic Tools were made available.
Mobile talk page edit completion rate (by account status)

In this impact analysis of the mobile Reply and New Topic Tools, we sought to answer two primary questions:

  1. After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would successfully publish the new talk page topics and comments they started drafting change?
  2. After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would publish talk page edits that are reverted within 48 hours of being published?.

To answer the questions above, we compared the rates at which people published the edits they started and the rates at which published edits were reverted before and after the mobile Reply and New Topic Tools were made available at the following Wikipedias: Arabic, Bengali, Chinese, French, Hebrew, Japanese, Haitian Creole, Korean, and Vietnamese. The results from this test can be found in the "Findings" section below.

Timing

This analysis was completed on 1 September 2022. These results include data from the two weeks before and after 29 June 2022.

A chart showing the changes in edit completion rate before and after the mobile Reply and New Topic Tools were made available.
Mobile talk page edit completion rate (by experience level)

Findings

  • Edit Completion Rate Increased
    • There was a 53.4% (12 percentage points) increase in the rate at which people who are logged in and logged out published the mobile talk page edits they started.
    • There was a 155% (14 percentage points) increase in the rate at which people who have made <100 cumulative edits published the mobile talk page edits they started.
  • Revert Rate Decreased
    A chart showing the mobile talk page edit revert rate before and after deployment for mobile reply and new topic tools
    Mobile talk pages revert rate (by account status)
    • There was a 5.6% decrease in the proportion of mobile talk page edits people who are logged in made that were reverted.
    • There was a 10.2% decrease in the proportion of mobile talk page edits people who are logged out made that were reverted.

Full report


Notifications[edit]

Analysis 2: Impact[edit]

In the A/B test of Topic Subscriptions, we sought to answer three primary questions:

  1. Do Topic Subscriptions cause contributors, across experience levels, to respond more quickly to new topics and comments that are posted on wikitext talk pages?
  2. Do Topic Subscriptions cause contributors, across experience levels, to receive responses to a greater percentage of the comments they post and new topics they start?
  3. Do Topic Subscriptions create unwanted noise that discourages people from using talk pages?

To answer the questions above, we ran an A/B test of Topic Subscriptions on desktop at 20 Wikipedias. The results from this test can be found in the "Findings" section below.

Timing

This analysis was completed on 5 August 2022. The A/B test ran from 2 June through 18 July 2022.

Findings

  • Topic Subscriptions increase the likelihood that volunteers who have made < 500 cumulative edits will receive a response to the comments they post.
    • There was a 8.6% increase in the percent of comments posted by people who have made under 100 cumulative edits that received a response
    • There was a 15.3% increase in the percent of comments posted by people who have made 100-500 edits that received a response
  • People who had Topic Subscriptions enabled respond to comments 57% faster (51 minute decrease in median response times) than people who did not have access to Topic Subscriptions
  • Topic Subscriptions reduce long lags in interaction
    • Among people who had access to Topic Subscriptions during the A/B test, 0.4% of responses were published >10 days after the initial comment was posted
    • Among people who did NOT have access to Topic Subscriptions during the A/B test, 7.3% of responses were published >10 days after the initial comment was posted
  • Topic Subscriptions did not cause a sharp increase in the number of notifications people receive each day
    • Prior to the A/B test, people at the participating Wikipedias received, on average, 3.6 notifications per day across
    • During the A/B test, people at the participating Wikipedias received, received, on average, 4.2 notifications per day
  • Topic Subscriptions did not cause fewer Senior Contributors to participate on talk pages.

Full Report

Topic Subscription AB Test Analysis

Analysis 1: Adoption[edit]

A bar graph showing how many distinct topics people have used the Topic Subscriptions feature to subscribe to.
The number of distinct topics people are actively subscribed to.

In this first analysis, we sought to learn the extent to which Senior Contributors are trying Topic Subscriptions (Manual and Automatic) and finding them valuable.

This information is necessary for the team to decide whether the impact of the feature is ready to be analyzed through an A/B test that is being planned in T280897.

Meta

Findings

Usage

  • 28% of all people (40% of Senior Contributors) that edited a talk page and have the DiscussionTools beta feature enabled have subscribed to at least 1 topic since 19 October 2021 (when manual topic notifications was available as a beta feature to all Wikis)
  • The majority (65%) of people actively subscribed to manual topic subscriptions are subscribed to 1 to 2 topics. 35% of manual topic subscribers have subscribed to more than 2 distinct topics.

Value

  • People value having the option to subscribe to topics: 99.5% of people who who received a new comment notification after manually subscribing to a discussion kept the feature enabled.
  • People read the new comment notifications they receive: people read 96% of the new comment notifications they receive within two weeks of receiving them.
  • No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
  • Some people are using Topic Subscriptions a lot: while the majority of people (65%) who have manually subscribed to a topic have subscribed to 1 to 2 distinct topics, 6% of people are actively subscribed to more than 20 topics.

Disruption

  • No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.


New Topic Tool[edit]

Analysis 2: Impact[edit]

In the A/B test of the New Topic Tool, we sought to learn two things:

  1. Does the New Topic Tool cause a greater percentage of Junior Contributors to publish the new topics they start without a significant increase in disruption?
  2. Does the New Topic Tool cause Junior Contributors to be more likely to start (activation) and continue (retention) participating on Wikipedia talk pages?

To answer the questions above, we ran an A/B test of the New Topic Tool from 27 January 2022 through 25 March 2022 on 20 Wikipedias. The results from this test can be found in the "Findings" section below.

Timing

This analysis was completed on 13 April 2022. The A/B test ran from 27 January 2022 through 25 March 2022.

Findings

  • The New Topic Tool causes logged-in Junior Contributors to have more success publishing topics:
    • A chart showing the percentage of Junior Contributors and Senior Contributors that attempted to start a new topic and were successful in publishing at least one of these new topics during the A/B test.
      The percentage of Junior Contributors and Senior Contributors that attempted to start a new topic and were successful in publishing at least one of these new topics during the A/B test.
      Junior Contributors who open the New Topic Tool are ~1.3 times more likely to successfully publish a new topic than Junior Contributors who open the existing add new section workflow.
    • 44.2% of Junior Contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 37.2% of Junior Contributors using the existing add new section workflow (+ 7 percentage points; 19% increase).
  • The New Topic Tool causes logged-in Junior Contributors to be less disruptive:
    A chart chowing the rates at which the edits Junior Contributors make with the New Topic Tool and the existing add section workflow are reverted.
    The rate at which the edits Junior Contributors make with the New Topic Tool and the existing add section workflow are reverted.
    • There was a 45% decrease (-1.8 percentage points: 4.05% → 2.22%) in the percent of Junior Contributors blocked after posting a new topic using the New Topic Tool.
    • There was an 11.6% decrease (-1.3 percentage points: 11.2% → 9.9%) in the percent of new topics reverted 48 hours after they were posted using the New Topic Tool; however, there is not sufficient evidence to confirm this observed change was due to the New Topic Tool.
  • Senior Contributors are more successful publishing topics using the existing add section workflow than they are using the New Topic Tool.
    • 66% of Senior Contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 71% of Senior Contributors using the existing add new section workflow (- 5 percentage points; 7% decrease).
  • The New Topic Tool led to a slight increase in the new topic completion rate for logged-out contributors; however, there is not sufficient evidence to confirm this observed change was due to the New Topic Tool.
    • 9.6% of logged-out contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 8.1% of Junior Contributors using the existing add new section workflow (+ 1.5 percentage points; 18.5% increase).
  • The New Topic Tool led to slight increases in the revert rate for new topic tool edits made by logged-out users and logged-in Senior Contributors; however, there is not sufficient evidence to confirm either of these observed changes were due to the New Topic Tool.
    • There was a 0.6 percentage point increase (1.5% → 2.1%; 40% increase) in the revert rate for new topic tool edits made by Senior Contributors compared to edits they made using the previous add section link.
    • There was a 3 percentage point increase (20.7% → 23.7%; 14.5% increase) in the revert rate for new topic tool edits made by logged-out users compared to edits they made using the previous add section link.

Reports

The full A/B test reports can be found here: Logged-In New Topic AB Test Report + Logged-Out New Topic AB Test Report.

Analysis 1: Adoption[edit]

In this first analysis, we sought to learn whether people who have the New Topic Tool enabled as a beta feature are finding the tool disruptive and behaving in the ways they expect. We were also curious to begin to understand how heavily people were depending on the tool to start talk page discussions.

To determine the extent to which people were finding the tool "disruptive" and "behaving in ways they expect," we looked at the following metrics:

  • The percent of distinct contributors that explicitly disabled the New Topic Tool after publishing at least one new topic with the tool.
  • The percent of all published edits made with the New Topic Tool that are reverted within 48 hours of being published compared to the percent of all talk page edits made with full-page editing that are reverted within 48 hours of being published.

To begin to understand how "heavily people were depending on the tool to start talk page discussions", we looked at the following metrics:

  • The percent of distinct contributors who published at least one new topic with the tool.
  • For contributors that have published at least 1 new topic with the New Topic Tool, the percent of distinct contributors that used the New Topic Tool to create the following percentage of all new topics: 0-25%, 26-50%, 51-75%, or 76-100% of new topics.

Timing

This analysis was completed on 17 August 2021.

Conclusions and Findings

  • Conclusion: The New Topic Tool is not disruptive to the majority of people who have used it as well as to the people who maintain the pages on which the tool is being used.
    • Finding: 91.68% of contributors that published at least one new topic with the New Topic Tool kept the tool enabled during the reviewed time period.
    • Finding: Of all the edits Junior Contributors made to talk pages during the time period that was analyzed, the proportion of reverted edits Junior Contributors made with the New Topic Tool was 21.3% less than the proportion of reverted edits they made with the existing, full-page editing interface.
  • Conclusion: Among people who started multiple discussions during the analysis period, nearly all of them came to depend on the New Topic Tool to start new topics.
    • Finding: Of the contributors that published more than one new topic on a talk page including at least one with the New Topic Tool, 95.3% of these contributors published between 75 to 100 percent of all their new topics using the New Topic Tool.

Report

You can review the full analysis results in this Jupyter notebook.


Reply Tool[edit]

Analysis 3: Impact[edit]

Percent of Junior Contributors that completed at least one comment attempt on a talk page during the AB test.
Percent of Junior Contributors that completed at least one comment attempt on a talk page during the AB test. | Source
The percent of comments made by Junior Contributors on talk pages that were reverted within 48 hours of being published.
The percent of comments made by Junior Contributors on talk pages that were reverted within 48 hours of being published. | Source

In this last analysis, we sought to learn two things:

  1. Does the Reply Tool cause Junior Contributors to be more successful publishing comments on talk pages?
  2. Does the Reply Tool cause more disruption to other volunteers?

To answer the questions above, we ran an A/B test of the Reply Tool from 11 February through 10 March on 22 Wikipedias. The results from this test can be found in the "Findings" section below.

Timing

This analysis was completed on 23-April-2021. The A/B test ran from 11-Febraury-2021 through 10-March-2021.

Conclusions

The clear evidence that shows the Reply Tool causes a greater percentage of Junior Contributors to publish a comment without a significant increase in disruption, signals to the team that it would benefit all Wikimedia Projects and volunteers to have the Reply Tool made available as an opt-out preference.

Findings

The data below included logged-in users who had not previously interacted with the Reply Tool (defined as users whose discussiontools-editmode preference was empty).

  • 72.9% of Junior Contributors who had access to the Reply Tool were able to publish at least one comment during the A/B test compared to 27.6% of Junior Contributors who did NOT have access to the Reply Tool.
  • Junior contributors were 7.2 times more likely to publish a comment than junior contributors using page editing.
  • There was a 79.5% decrease in the revert rate of comments Junior Contributors made with the Reply Tool compared to comments Junior Contributors made with page editing.

Report

You can review the full analysis results here: https://wikimedia-research.github.io/Reply-tools-analysis-2021//.

Analysis 2: Engagement[edit]

The next analysis we did was needed to help determine whether people, across experience levels, were having success using the Reply Tool and whether people were using the tool in ways that degrades the experiences of others.

This analysis's findings were used to decide whether the Reply Tool is functioning well enough for its impact on user behavior to be tested via a larger-scale (read: at wikis beyond our partner wikis) A/B test.

Timing

This analysis was completed on 5-January-2021

Conclusions

The high rate at which people who used the Reply Tool to publish the comments they started writing, combined with the lack of clear evidence of disruption, led the team to think the Reply Tool was ready to be tested at a larger scale via an A/B test. More details below.

Findings

  • The edit completion rate for Junior Contributors using the Reply Tool is 4x higher than the edit completion rate for people using full-page wikitext editing. Source
  • People who have made <1,000 cumulative edits, are between 32% and 510% more likely to publish edits with the Reply Tool (across modes) than they are with full page wikitext talk page editing. Source
  • <1% of comments Junior Contributors post with the Reply Tool are reverted within 48 hours, across wikis (this number is higher for Senior Contributors ). Source

Analysis 1: Adoption[edit]

In this first analysis, we sought to learn whether people at the Arabic, Dutch, French and Hungarian Wikipedias, where the Reply Tool had been available as a Beta Feature since 31-March-2020, were finding the tool valuable.

To determine the extent to which people were "finding the tool valuable" we looked at how frequently people were using the tool (as measured by the number of distinct days they use it on) and how intensely people were using the tool (as measured by the total number of edits they make with the Reply Tool and the percentage of total talk page edits they used the Reply Tool to make).

Understanding the above helped us determine the tool was ready to be made available as an opt-out feature at these wikis and made available as an opt-in Beta Feature at others.

Timing

This analysis was completed on 28-July-2020.

Findings

The data below included people who have used the Reply Tool as a Beta Feature at the Arabic, Dutch, French and Hungarian Wikipedias, between 31-March and 30-June-2020:

  • 328 people posted at least one comment using the Reply Tool
  • 69.5% of people posted ≥ 2 comments with the Reply Tool
  • 61.3% of people posted a comment using the Reply Tool on ≥ 2 distinct days*
  • 23.6% of people used the Reply Tool to make over half of their talk page edits**

Report

You can review the full analysis results in this Jupyter notebook: https://nbviewer.jupyter.org/github/wikimedia-research/Reply-tools-analysis-2021/blob/master/Adoption-Metrics/Replying-Tool-Adoption-Metrics.ipynb.

*A "distinct day" was defined as a distinct calendar day. Therefore, some of these edits may have occurred 24 hours apart and some may have occurred only a few hours apart depending on what time of day the edit was made.

**This data is a bit noisy in so far as there are some edits that cannot be made with the Reply Tool (e.g. starting a new section, editing existing content, etc.) and the software is not currently able to differentiate between comments and other talk page edits.