Edit check/Reference Check

Wikimedia Foundation projects Reference check Prompt newcomers to reference information they are adding to Wikipedia. Group: Editing Start: 2023-02-01 Team members: David Chan, David Lynch, Esther Akinloose, Marielle Volz, Rummana Yasmeen, Nicolas Ayoub, Benoît Evellin, Ed Sanders, Zoe, Caro Medelius Backlog: #EditCheck Management: Peter Pelberg (product), Valerie Puffet-Michel (engineering)

Reference Check is an Edit check that prompts newcomers to add citations before they publish an edit that adds new text to a Wikipedia article.

This Check is an effort to increase the likelihood that the new content people are adding to Wikipedia is aligned with WP:Verifiability.

To participate in and follow this project's development, we recommend adding this page to your watchlist

Status

As of September 2025, Reference Check is available by default at all Wikipedias except English Wikipedia .

Objectives

Reference Check aims to:

1. Cause newcomers to publish new content edits that include references while lowering the likelihood those edits are reverted
2. Reduce the amount of effort and attention experienced editors need to allocate to ensuring text in Wikipedia's namespace is accompanied by a reliable source

Background

Accompanying additions to Wikipedia with citations is an important part of Wikipedia's five pillars.

Accompanying additions to Wikipedia with citations is also a practice many newcomers do not follow.

Proportion of new content edits that include a new reference

A July 2023 analysis found that 17% of the new content edits newcomers (≤100 cumulative edits) publish using the visual editor include a new reference.

With the above in mind, Reference Check is meant to address two core issues:

1. Newcomers publishing new text to Wikipedia that does not include a reference
2. Experienced volunteers being burdened by the effort and attention they need to allocate towards finding sources to verify the reliability of the information available on Wikipedia. This effort can come at the expense of identifying and addressing more subtle and complex forms of undesirable editing.

Reference Check is designed to address these two issues by:

1. Offering new(er) volunteers feedback while they are editing so they can avoid unintentionally publishing edits that are likely to violate policies
2. Offering patrollers/reviewers deeper insight into the edits they are reviewing (and the intentions of the people publishing them) by logging the moderation feedback newcomers are being presented with, and the actions they do/do not take in response.

Design

User Experience

The initial version of the Reference Check experience will prompt people to decide whether they want to add a reference or not.

Reference Check (desktop)

This prompt will be presented to people who add a minimum of one new paragraph of text when they tap "Publish changes..." without accompany this new text with a reference.

Reference Detection

For the first version of Reference Check, the Editing Team is pursuing an approach that minimizes the likelihood of false positives and is implemented in ways that empower volunteers, on a per-project basis, to evolve the detection heuristic to become more robust over time.

This strategy amounts to the initial Reference Check becoming activated if/when all of the following conditions are met:

1. A minimum of one new paragraph of text is added to the article someone is editing
2. The "new paragraph(s) of text" someone has added does NOT include a reference
3. The changes described in "1." and "2." are happening on a page within the main namespace (NS:0)

The conditions above are implemented and maintained in code here: editcheck/init.js.

The Editing Team arrived at the decision to start with a relatively limited and straightforward set of rules in order to:

1. Increase the likelihood that newcomers find the guidance Reference Check is presenting them with, and the editing experience more broadly, to be intuitive and straightforward so that they feel encourage to return to edit again
2. Decrease the likelihood that Reference Check is creating more work for experienced volunteers by prompting newcomers to add sources when they are not needed

You can learn more about the assumptions that informed the thinking above in phab:T329988#8654867.

Configurability

Reference Check is implemented – like all Edit Checks – in a way that enables volunteers to explicitly configure how it behaves and who the Check is made available to.

Configurability happens on a per project basis so that volunteers can ensure the Reference Check experience is aligned with local policies and conventions.

Experiments

A/B Test (English Wikipedia)

Planning is underway to evaluate the impact of Reference Check at English Wikipedia through a dedicated A/B test.

Multi-Check (References) A/B Test

Multi-Check includes a series of changes to the visual editor user experience that enables multiple Edit Checks, of the same or different types, to be presented within a single edit session.

To learn whether these changes were effective at causing, newcomers to publish new content edits that include references while lowering the likelihood those edits would be reverted, we ran an A/B test with 12 Wikipedias.

Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.

Conclusion and next steps

Reference likelihood based on number of Reference Checks shown

Multi-Check caused statistically significant increases in the proportion of edits that include a reference and are not reverted within 48 hours without causing undesirable changes in edit completion or other forms of disruption (e.g. blocks).

In response, Multi-Check was deployed to all wikis where Reference Check is currently available on June 25, 2025.

Findings

• Proportion of new content edits with a reference
  • Users are more likely to include at least one reference with their new content edits when multi-check (references) is available.
    • Edits that were shown multiple Reference Checks in a session are 1.3 times more likely to include at least one new reference in the final published edit compared to sessions shown a single Reference Check.
• Revert Rate
  • Overall, we did not identify any significant changes in new content edit revert rate between the control and test group.
  • However, there was a -34.7% decrease in revert rate when directly comparing edits presented multiple checks compared to edits presented a single reference check
• Proportion of users that publish at least one new content edit with a reference
  • Overall, there was a 5.5% increase in the proportion of distinct users who published a new content edit with a reference when multi-check was available.
• Guardrails
  • There were no decreases in edit completion rate for up to 5 reference checks being presented in a single session (which accounts for the majority of multi-check edits).
  • We also did not identify any increases in revert rate at any number of reference checks presented.

Leading Indicators (T388731)

• New(er) volunteers are encountering Multi-Check
  • In the test group, multiple reference checks were shown within a single editing session at 19% of all published new content VE edits (549 edits) by unregistered users and users with 100 or fewer edits.
  • For edits shown multiple checks, the majority of edits (73%) were shown between 2 to 5 Reference Checks.
• People shown multiple Reference Checks w/in an edit go on to publish at a relatively high rate
  • The edit completion rate for sessions that were shown multiple checks within a session was 76.1% compared to 75% for sessions shown only one check, indicating that multiple checks are not causing significant disruption or confusion to the editors.
  • Likelihood to include a reference
    • Sessions shown multiple checks are more likely to include at least one new reference in the final published edit compared to sessions shown just a single check.
    • In the test group, 52.5% of all published edits shown multiple Reference checks included at least one new reference compared to 39.7% of edits that were shown a single check.
  • Disruption (revert and block rates)
    • In the test group, the revert rate of new content edits shown multiple Reference Checks (17%) is currently lower compared to sessions shown a single Reference Check (26%).
    • No significant changes in the proportion of users blocked after being shown multiple Reference Checks compared to a single Reference Check.

Multi-Check Phase 1: Impact Analysis

Multi-Check (Phase 1)

In December 2024, we released a new design for the Edit Check desktop experience. This change shifted Edit Checks from appearing within articles to appearing alongside them, in a new "siderail."

To decide how – if at all – this change impacted volunteer disruption and edit quality, we compared analyzed several key metrics shifted before and after this change.

Findings

• The revert rate of new content edits where Reference Check was activated decreased by 15.7%.
  • 20.4% (pre) →  17.2% (post).
• The proportion of new content edits that included a reference following the change increased 8%.
  • 34.8% (pre) → 37.9% (post)
• Excluding reverted edits, there was 3.2% increase [2 percentage points] in edit completion rate.
  • 68% of edits where Reference Check was presented were successfully saved and not reverted following the change in the Edit Check UX.
• The rate at which people declined to add a reference when Edit Check prompted them decreased 4.7%.

Conclusion(s)

The findings above suggest this design change has been net positive. As a result, we will continue depending on this new design paradigm as we introduce new types of Edit Checks and moments within the editing workflow when they are presented.

Reference Check A/B Test

To learn whether the Reference Edit Check is effective at causing newcomers to make edits they intended and experienced volunteers value , we conducted an A/B test with 15 Wikipedias.

Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.

Conclusion and next step(s)

Reference Check caused an increase in the quality of edits newcomers publish and did not cause any significant disruption.

This combination is leading the Editing team to be confident that offering Reference Check as a default-on feature would have a net positive impact on all wikis and the people who contribute to them.

You can read the full A/B test report here.

Findings

There was 2x increase in the proportion of new content edits by newcomers, Junior contributors, and unregistered users that included a reference when Reference Check was shown to eligible edits.

New content edits *with* a reference

People shown the Reference Check are 2.2 times more likely to publish a new content edit that includes a reference and is constructive (not reverted within 48 hours).

• Increases were observed across all reviewed user types, wikis, and platforms.

• The highest observed increase was on mobile where contributors are 4.2 times more likely to publish a constructive new content edit with a reference when Reference Check was shown to eligible edits.

Revert rate

• New content edit revert rate decreased by 8.6% if Reference Check was available.
  • New content edits by contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.

We observed increases on both desktop and mobile. On mobile, users are 4.2 times more likely to include a reference with their new content when the Reference Check is shown to eligible edits.

While some non-constructive new content edits with a reference were introduced by this feature (5 percentage point increase), there was a higher proportion of constructive new content edits with a reference added (23.4 percentage point increase). As a result, we observed an overall increase in the quality of new content edits.

There was a -8.6% decrease in the revert rate of all new content edits comparing edits where Reference Check was shown in the test group to edits that were eligible but not shown Reference Check in the control group.

Constructive Retention Rate

• Contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month (31-60 days after).
  • This increase was primarily observed for desktop edits. There was a non-statistically significant difference observed on mobile.

Guardrails

Edit Completion Rate

• We observed no drastic decreases in edit completion rate from intent to save (where Reference Check is shown) to save success overall or by wiki.
• Overall, there was a 10% decrease in edit completion rate for edits where Reference Check was shown.
  • There was a higher observed decrease in edit completion rate on mobile compared to desktop. On mobile, edit completion rate decreased by -24.3% while on desktop it decreased by -3.1%.

Block Rate

• There were decreases or no changes in the rate of users blocked after after being shown Reference Check and publishing an edit compared to users in the control group.

False Negative Rate

• There was a low false negative rate. Only 1.8% of all published new content edits in the test group did not include a new reference and were not shown Reference Check.

False Positive Rate

• 6.6% of contributors dismissed adding a citation because they indicated the new content being added does not need a reference. This was the least selected decline option overall.

Test design

11 Wikipedias participated in the test. At each wiki, 50% of users were randomly assigned to a test group and 50% were assigned to a control group.

Users in the test group were shown the Reference Check notice prompting them to decide whether the new content they were adding need a reference (if they had not already added one themselves).

User in the control group were shown the default editing experience, even if they did not accompany the new content they were adding with a reference.

Timing

This analysis was completed on 16 April 2024 and analyzed engagement data at the 11 participating wikis from 18 February 2024 through 4 April 2024.

Edit Check

This initiative sits within the larger Edit check project – an effort to meet people while they are editing with actionable feedback about Wikipedia policies.Edit Check is intended to simultaneously deliver impact for two key groups of people.

Experienced volunteers who need:

1. Relief from repairing preventable damage
2. Capacity to confront complexity

New(er) volunteers who need:

1. Actionable feedback
2. Compelling opportunities to contribute
3. Clarity about what is expected of them