Jump to content

Vérification des modifications/Vérification du ton

From mediawiki.org
This page is a translated version of the page Edit check/Tone Check and the translation is 99% complete.

Cette page regroupe le travail fait par l'Equipe édition en collaboration avec l'Equipe apprentissage automatique pour développer le Contrôle du ton (anciennement Peacock Check).[1]

Le contrôle du ton fait partie du contrôle des modifications. Il utilise un modèle de langue qui s'affiche aux rédacteurs lorsqu'ils emploient dans leurs ajouts un langage promotionnel, abject, ou subjectif d'une certaine manière, afin qu'ils reformulent leur texte de manière plus neutre.

Aspect remarquable de ce projet : le contrôle du ton est le premier contrôle des modifications qui utilise l'apprentissage automatique (machine learning). Pour ce cas, un modèle BERT de langage est sélectionné initialement et adapté précisément par l'équipe Recherche pour identifier les langues biaisées dans le nouveau texte que les éditeurs essaient de publier sur Wikipedia.

Il est possible de tester le contrôle du ton sur divers wikis .

La page d'aide du contrôle des modifications documente la manière dont le contrôle du ton fonctionne et peut être utilisé.

Pour participer et suivre le développement de ce projet, nous vous recommandons d'ajouter cette page à votre liste de suivi.

État

Dernière mise à jour :

Travail en cours actuellement

  1. Examiner comment le contrôle du ton pourrait aider les patrouilleurs et les relecteurs à détecter plus facilement les formes subtiles de vandalisme.
  2. Déploiement du contrôle du ton sur les Wikipedia françaises, japonaises et portugaises, suite aux résultats positifs de l'expérience contrôlée.
  3. Évaluation de l'efficacité de la balise editcheck-tone-shown

Occasions de proposer un commentaire

  1. Patrouilleurs et relecteurs : de quelle manière le contrôle du ton pourrait aider les patrouilleurs et les relecteurs à rendre plus facile la détection des formes subtiles de vandalisme ?voir la conversation.
  2. Configurabilité : quels seront les aspects configurables du contrôle de ton sur le wiki ? T393820

Calendrier

Expansion de la couverture de langues du modèle de contrôle du ton

Voir cette section du contrôle des modifications pour comprendre plus finement l'origine des développements.

Objectifs

La vérification du ton est destinée simultanément à :

  1. faire que les nouveaux bénévoles agissant de bonne foi pour ajouter de nouvelles informations dans l'espace de noms principal de Wikipedia, les écrivent dans un ton neutre.
  2. réduire l'effort et l'attention que les bénévoles expérimentés doivent déployer pour s'assurer que le texte de l'espace de nom principal est écrit dans un ton neutre.

Contexte

Ecrire d'un ton neutre est une partie importante de la Politique du point de vue neutre de Wikipedia.

En outre, beaucoup de nouveaux bénévoles trouvent que l'écriture dans un ton neutre est une pratique peu intuitive. Une analyse d'octobre 2024 des modifications de nouveau contenu que les bénévoles nouveaux venus[2] ont publié sur la Wikipedia anglaise a conclu que :

  • 56% des modifications de nouveau contenu publié par les nouveaux bénévoles contenait un ton qui n'était pas neutre.
  • 29% des modifications de nouveau contenu publié par les nouveaux bénévoles contenaient un ton qui n'était pas neutre, et ont été annulées.
  • les nouvelles modifications de contenu contenant des mots non neutres avaient 46.7% de chances d'être annulées par rapport aux modifications de nouveau contenu avec un ton neutre

Suite à ce qui vient d'être dit, le contrôle du ton tente de résoudre deux problèmes :

  1. Les nouveaux venus qui publient des modifications dans Wikipedia utilisant un langage promotionnel, abject, ou autrement subjectif à cause de leur manque de connaissance à propos du suivi des politiques de Wikipedia.
  2. Les bénévoles expérimentés à qui on demande un effort supplémentaire et l'attention qu'ils doivent porter au patrouillage et à la relecture pour empêcher les dommages faits de bonne foi. Ceci peut aboutir à la dépense pour identifier et corriger des formes plus subtiles et complexes de vandalisme. Le contrôle du ton est conçu pour répondre à ces deux problèmes en :
    1. affichant aux nouveaux bénévoles des messages pendant qu'ils modifient de sorte à leur éviter de publier malencontreusement des modifications qui violeraient les politiques
    2. en offrant aux patrouilleurs et aux relecteurs une vue plus profonde des modifications relues (et les intentions des utilisateurs qui les publient) en journalisant les commentaires de modération présentés aux nouveaux rédacteurs, et les actions qu'ils prennent (ou ne prennent pas) en réponse.

Architecture

Screenshot showing the proposed Tone Check card on mobile.
Screenshot showing the proposed Tone Check card on mobile.

Expérience utilisateur

Tone Check is a contextual intervention designed to equip new(er) volunteers editing in good faith with the awareness and know-how needed to  ensure the tone of  the text they are adding is aligned with Wikipedia policies. The Check is intentionally minimal, appears only when necessary, and aims to support the policies wikis have individually defined without blocking contributions.

When Tone Check is shown

Tone Check becomes activated when a contributor (who meets the configuration criteria communities will be able to set) adds new text that the underlying machine learning modeltrained on Wikipedia edits – identifies as potentially biased or promotional. Specifically:

  • The check activates after the user finishes editing a paragraph and clicks or taps outside of it.
  • If the system detects promotional, derogatory, or otherwise subjective language, the Tone Check card appears.

The Edit Check card is displayed in the side container on both desktop and mobile devices. This lightweight, non-blocking format allows contributors to stay in flow while being made aware that something they have done warrants additional attention.

Placement and interaction

Tone Check appears at two key points in the editing workflow:

  • Mid-edit: If detected while the contributor is actively editing, Tone Check appears immediately in the side panel (mobile and desktop)
  • Pre-save: If not acted upon earlier, Tone Check appears again after the contributor clicks or taps “Publish changes”, during the proofreading step.
Screenshot showing the first iteration of the Tone Check card
Edit Check card

When Tone Check is activated, contributors see an Edit Check card with:

  • A short explanation that the flagged language is often revised by other editors for a more balanced tone.
  • A “Learn more” link to access additional context about Wikipedia’s tone policies and guidelines.
  • Two actions:
    • Revise, return to editing and update the highlighted text.
    • Decline, proceed as-is, after selecting a reason for not revising.
  • A disclaimer noting that a small language model was used to detect tone-related issues in the text.

Design intent and principles

Tone Check is grounded in the following design principles:

  • No firm rules: The tool suggests, but does not force, changes. Contributors can always choose to decline or proceed without editing.
  • Keep users in flow: The experience is embedded within the natural flow of editing and publishing, with lightweight prompts that avoid blocking or clutter.
  • Meet users where they are: Feedback is specific to the paragraph being edited and is framed using language easy to understand and grounded in Wikipedia norms.
  • Transparent: A clear disclaimer and consistent design patterns ensure transparency about how suggestions are generated.

Sélection de langue

This section to include the languages we're prioritizing for initial experiment, the languages we're planning to scale to next, and why we came to select these languages. See phab:T388471.

Modèle

Tone Check leverages a Small Language Model (SLM) to detect the presence of promotional, derogatory, or otherwise subjective language. The SLM we are using is a BERT model, which is open source and presents its weights openly.

The model works by being fine-tuned on examples of Wikipedia revisions. It learns from instances where experienced editors have applied a specific template ("peacock" and equivalent templates) to flag tone violations, as well as instances where that template was removed. This process teaches the BERT model to identify patterns associated with appropriate and inappropriate tones based on Wikipedia's editorial standards. Under the hood, SLMs work by transforming text into high-dimensional vectors, which are then compared with the label, allowing the model to find a hyperplane that splits text into negative or positive cases.

The model was trained using 20,000 data points from 10 languages consisting of:

  • Positive examples: Revisions on Wikipedia that were marked with the "peacock" template, indicating a tone policy violation.
  • Negative examples: Revisions where the "peacock" template had been removed (signifying no policy violation).

Small Language Models (SLMs — like the one being used for Tone Check) differ from Large Language Models (LLMs) in that the former are trained to adapt for particular use cases by learning from a focused dataset. In the case of Tone Check, this means the SLM learns directly from the expertise of experienced Wikipedia volunteers. Hence, they offer more explainability and flexibility compared to LLMs. Also SLMs requires significantly fewer computational resources than its larger counterparts.

LLMs on the other hand, are designed to work for general-purposes, with limited context and through a chat or prompting interface. LLMs require a huge amount of computation resources, and their behavior is difficult to explain, due the high amount of parameters involved.

Evaluation du modèle

Deux évaluations du modèle

Before measuring the impact of the overall Tone Check experience through a controlled experiment in production, the team conducted two evaluations comparing the model's predictions to human-provided labels.

Outlined below is information about the purpose of each evaluation and what we found.

Evaluation interne

Goals

The first evaluation we conducted was internal, involving just the WMF product teams who were working on this feature. This review was meant to:

  1. Evaluate whether the model aligned with human decisions often enough that we could consider its predictions reliable enough to move forward with a community-involved evaluation process
  2. Figure out a prediction probability score threshold above which we could consider the model's predictions fairly accurate
  3. Expose any edge cases or specific types of edits in which the model consistently does not perform well

Process

To assess the above, the team:

  1. Created a list of 300 sample edits from English Wikipedia.
  2. Assigned about 30 edits to each of the participants from our teams.
  3. Asked each participant to go through the sample edits and indicate whether or not they contained promotional, derogatory, or otherwise subjective language that should be flagged by the Tone Check.
  4. Compared the model's predictions to the human-provided labels.
  5. Analyzed the cases where the model's predictions differed from the human-provided labels.

Findings

  1. In English, false negatives (cases where the model predicts there isn't a tone check issue, but a human says there is) are very easily filtered out if we only return predictions with a probability score over 0.55.
  2. In English, most false positives (cases where the model predicts that there is a tone check issue, but a human says there isn't) can be filtered out if we only return predictions with a probability score over 0.8.
  3. There are some types of edits that the model has a hard time with - like edits that include a quote, where the quoted language is non-neutral in tone. In these cases, the model's predictions had a lower probability score.

Evaluation des bénévoles

Screenshot showing an example of a diff volunteers used to review the predictions the Tone Check model makes.
An example of a diff volunteers used to review the predictions the Tone Check model makes.

The results of the internal evaluation gave us confidence to move forward with an external review involving experienced volunteers. We had enough positive examples (as defined above) to continue evaluating the model in English, French, Japanese, Portuguese, and Spanish.

Goals

This second review meant to:

  • Help us confirm that experienced volunteers agree with what the model identifies as promotional, derogatory, or otherwise subjective language
  • Evaluate whether the model's predictions about edits in French, Japanese, Portuguese, and Spanish are as reliable as they are about edits in English

Process

To assess the above, the team:

  1. Created a list of 100 sample edits from each of the aforementioned Wikipedias.
  2. Invited participants from each Wikipedia community to sign up and participate.
  3. Provided the participants with a tool they could use to review and label each of the sample edits in the language(s) they were helping with.
  4. Asked each participant to review and label at least 30 sample edits.
  5. Compared the model's predictions to the human-provided labels.
  6. Analyzed the cases where the model's predictions differed from the human-provided labels.

Findings

At the probability threshold the model would need to reach for a Tone Check to be shown (0.80) during an edit session, volunteers across the 5 languages who participated in the initial model review agreed with the model's detection of a tone issue in 95% of cases.

More details about the results from each of the 5 languages that we included in the initial volunteer review can be found in the table below.

Language Revisions reviewed Unique participants High-level findings Recommendation How we made this recommendation
English 391 13
  • 5% of reviews were false positives
  • No false positives with a probability above 0.67
Continue conversations with volunteers about the potential risks of the feature and ideas for how we might mitigate and manage them. It was rare for the model to flag an edit for a tone issue when volunteer reviewers said there wasn’t one - this only happened about 5% of the time. When it did happen, the model wasn’t very confident in its prediction. Its probability score was below the threshold  (0.8) that we’d use in a real-world setting.
Spanish 285 9
  • 3% of reviews were false positives with a probability of 0.8 or above and there were many “false positive” cases where the added text did actually contain biased language.
  • 2 samples (with 3 total reviews) were positive for unclear reasons that needed to be investigated (probability 0.87 and 0.82 probability). Both were samples where only a phrase was added to a paragraph, and the added phrase did not contain non-neutral language.
Proceed with 0.8 probability score threshold and recommend es.wiki evaluates feature through an A/B test. Most of the time, when the model confidently (with a probability score of ≥0.8) flagged an edit for a tone issue, the volunteers who reviewed that edit agreed - it was indeed a problem.

In the small number of cases (3%) where the model confidently flagged an issue but at least one volunteer disagreed, there wasn’t a clear consensus among the volunteer reviewers. Even then, most volunteer reviewers still sided with the model. These cases often involved subjective or opinionated phrases, like “uno de los doctores mas importantes” (“one of the most important doctors”) and “desarrollo un paupérrimo torneo” (“he had a very poor tournament”).

Japanese 228 5
  • Fewer high-probability predictions overall, compared to other languages. Only 3% of samples saw probability over 0.8.
  • Higher proportion of false positives (27%) but none had a probability of 0.8 or above.
  • Two false positives had a probability score of 0.7 or above and in both cases, ≥1 other volunteer reviewers agreed with the model.
Proceed with 0.7 probability score threshold* and propose ja.wiki evaluates feature through an A/B test

*This recommendation assumes ja.wiki is generally open to Tone Check; if it is more conservative, recommend higher threshold to minimize false positives

The model didn’t make very many high-confidence predictions for Japanese - only 3% of predictions had a probability score above 0.8, which was the threshold we had planned to use in the production experiment.

Because so few predictions reached that level of confidence, we recommend lowering the threshold to 0.7 for Japanese. Importantly, none of the predictions with a score above 0.8 flagged a tone issue in edits that volunteers thought were fine.

There were two cases where the model predicted a tone issue at probability scores of 0.7 and 0.75, and at least one human reviewer disagreed. But in both cases, there was no clear agreement among the reviewers - at least one reviewer agreed with the model’s assessment.

Portuguese 22 2
  • More reviews required for results to be conclusive
  • No false positives (out of 22 reviews)
  • All model predictions were above 0.8 probability or below 0.69 probability.
Proceed with 0.8 probability score threshold and propose pt.wiki evaluates feature through an A/B test. In parallel, recruit more volunteers to review model. We only received 22 reviews, which wasn’t enough for a thorough evaluation. Of the edits reviewed, 50% had a model probability score above 0.8 - so we’re not too worried about recall.

Additionally, in the cases where the model predicted a tone issue, human reviewers always agreed, meaning there were no false positives.

French 369 6
  • 8% of reviews were false positives.
  • 4% of reviews were false positives with a model probability score of 0.8 and above.
  • In all the false positives with probability above 0.75, there were no examples where volunteers unanimously disagreed with the model.
Proceed with 0.8 probability score threshold and propose fr.wiki evaluates feature through an A/B test Most of the time, when the model confidently flagged an edit for a tone issue, volunteers agreed - it was indeed a problem.

In the small number of cases (4%) where the model confidently flagged an issue but at least one volunteer disagreed, there wasn’t a clear consensus among the human reviewers. Even then, many reviewers still agreed with the model. These cases often involved subjective or opinionated phrases, like “sa démarche picturale novatrice, paradoxale et indépendante” (“his innovative, paradoxical and independent approach to painting”).

Expérience utilisateur

The viability of Tone Check, like the broader Edit Check project, depends on the feature being able to simultaneously:

  1. Reduce the moderation workload experienced volunteers carry
  2. Increase the rate at which new(er) volunteers contribute constructively

To evaluate the extent to which Tone Check is effective at the above, the team will be conducting qualitative and quantitative experiments.

Below you will find:

  1. Impacts the features introduced as part of the Edit Check are intended to cause and avert
  2. Data we will use to help[3] determine the extent to which a feature has/has not caused a particular impact
  3. Evaluation methods we will use to gather the data necessary to determine the impact of a given feature
Desired Outcomes
ID Outcome Data Evaluation Method(s)
1. Key performance indicator: The quality of new content edits newcomers and Junior Contributors make in the main namespace will increase because a greater percentage of these edits will not contain peacock language
  1. Proportion of all new content edits published without biased language
  2. Proportion of new content edits that are not reverted.
A/B test, qualitative feedback (e.g. talk page discussions, false positive reporting)
2. Key performance indicator: Newcomers and Junior Contributors will experience Peacock Check as encouraging because it will offer them more clarity about what is expected of the new information they add to Wikipedia Proportion of new content edits started (defined as reaching point that peacock check was or would be shown) that are successfully published (not reverted). A/B test, qualitative feedback (e.g. usability tests, interviews, etc.)
3. New account holders will be more likely to publish an unreverted edit to the main namespace within 24 hours of creating an account because they will be made aware the new text they're attempting to publish needs to be written in a neutral tone, when they don't first think/know to write in this way themselves Proportion of newcomers who publish ≥1 constructive edit in the Wikipedia main namespace on a mobile device within 24 hours of creating an account (constructive activation). A/B test
4. Newcomers and Junior Contributors will be more aware of the need to write in a neutral tone when contributing new text because the visual editor will prompt them to do so in cases where they have written text that contains peacock language. The proportion of newcomers and Junior Contributors that publish at least one new content edit that does not contain peacock language. A/B test
5. Newcomers and Junior Contributors will be more likely to return to publish a new content edit in the future that does not include peacock language because Peacock Check will have caused them to realize when they are at risk of of this not being true.
  1. Proportion of newcomers and Junior Contributors that publish an edit Peacock Check was activated within and successfully return to make an unreverted edit to a main namespace during the identified retention period.
  2. Proportion of newcomers and Junior Contributors that publish an edit Peacock Check was activated within and return to make a new content edit without non-neutral language to a page in the main namespace during the identified retention period.
A/B test
Undesirable Outcomes
ID Outcome Data Evaluation Method(s)
1. Edit quality decreases Proportion of published edits that add new content and are still reverted within 48hours. Note: Will include a breakdown of the revert rate of published new content edit edits with and without non-neutral language. A/B test and leading indicators analysis
2. Edit completion rate drastically decreases Proportion of new content edits started (defined as reaching point that peacock check was or would be shown) that are published. Note: Will include breakdown by the number of checks shown to identify if lower completion rate corresponds with higher number of check shown. A/B test and leading indicators analysis
3. Edit abandonment rate drastically increases Proportion of edits that are started (event.action = init) that are successfully published (event.action = saveSuccess). A/B test and leading indicators analysis
5. People shown Tone Check are blocked at higher rates Proportion of contributors blocked after publishing an edit where Tone Check was shown compared to contributors not shown the Tone Check A/B test and leading indicators analysis
6. High false positive rates Proportion of contributors that decline revising the text they’ve drafted and indicate that it was irrelevant. A/B test, leading indicators analysis, and qualitative feedback

Résultats

Expérience A/B

See full report.

Conclusions

A 2026 analysis of the Tone Check A/B experiment ran on French, Japanese, and Portuguese Wikipedias showed:

  1. Tone Check successfully decreases the frequency of non-neutral language in published content.
  2. Tone Check successfully decreases the likelihood that new content edits are likely to be reverted.
    1. Note Tone Check has an even stronger effect when people engage with the Check's prompt to revise the tone of what they've written.
  3. Tone Check successfully increases the constructive edit rate.
  4. Tone Check causes people to be more likely to return and publish a constructive edit within 2 weeks of making their first.
  5. Tone Check does not appear to be causing any significant disruption to most people’s editing experience.
  6. The feature did not cause any meaningful regressions in the guardrail metrics we were monitoring.

Overall (across platforms and experience levels) the A/B experiment demonstrated that Tone Check is effective at A) increasing the quality of new content edits newcomers and Junior Contributors publish and B) increasing the likelihood that they will return to publish a constructive edit within 2 weeks. Both of these effects were proven without negatively affecting the overall health of the edit funnel.

All of the above is causing the team to move forward with scaling Tone Check to all Wikipedias as the model's language support expands. See scenario 3 in T387918.

Résultats

  1. Bar graph showing rates at which published new content edits shown, or eligible to be shown, Tone Check are constructive.
    Overall, constructive edit rates increased by +6.2% increase [4.4 percentage points] for people shown Tone Check in the test group.
    Constructive edit rate
    1. Overall: Tone Check improved the rate of constructive edits by +6.2% [4.4] percentage points. We observed improvements in overall edit quality at each of the three partner Wikipedias.
    2. Platform: on desktop, constructive edit rate increased by +6.4% while we observed no statistically significant change in mobile web constructive edits.
    3. Experience level: Tone Check appears especially effective at increasing the constructive edit rate of a registered Junior Contributors, where we observed a +14.8% increase [10.2 pp] in constructive edit rates.
  2. New content edits published without biased language
    1. Overall: Tone Check successfully decreases the frequency of non-neutral language in published content
      1. Users with access to Tone Check were -15.6% less likely to publish edits containing non-neutral language (falling from 9.6% to 8.1%; a -1.5 pp decrease) compared to the control group.
        1. Note: we have 99.8% confidence that this improvement is directly attributable to the tool.
    2. Platform: trends differ.
      1. Results confirm a highly significant impact on Desktop, where we observed the highest reduction in revert rate. In contrast, there was no detectable effect yet on Mobile Web.
  3. Chart showing that, across mobile and desktop, there was a -15% decrease [-4.4 ppp] in the revert rate of edits shown Tone Check in the test group compared to edits eligible but not shown Tone Check in the control group.
    Across desktop and mobile, there was a -15% decrease [-4.4 ppp] in the revert rate of edits shown Tone Check in the test group compared to edits eligible but not shown Tone Check in the
    New content revert rate
    1. Overall: edits made by users shown Tone Check are also -15% less likely to be reverted than eligible control edits (29.5% → 25.1%; a -4.4 pp decrease).
    2. Experience level: we see the strongest effect among Junior Contributors.
      1. Among Junior Contributors, we observed a -33% relative [-10.2 pp] decrease in the rate at which the new content edits they publish are reverted. This finding was statistically significant.
      2. Among newcomers and unregistered users we did not confirm any statistically significant changes in the rate at which the new content edits they publish are reverted.
  4. Chart showing that when the Tone Check successfully prompts a user to remove non-neutral language, the likelihood of that edit being reverted drops -44.1%.
    When the Tone Check successfully prompts a user to remove non-neutral language, the likelihood of that edit being reverted drops -44.1%.
    Impact of removing non-neutral language
    1. Overall: when someone removes non-neutral language in response to a Tone Check, the likelihood of that edit being reverted decreased by -44.1%.
    2. Platform: while we observed statistically significant decreases on both platforms, the impact was higher on desktop.
      1. Desktop: we observed a significant -47% decrease [-13.4 pp] in revert rate for people who revised their text in response to Tone Check.
      2. Mobile: we observed a significant -14.8% [-4.8pp] decrease in revert rate for edits where non-neutral language was removed.
  5. Edit completion rate
    1. Overall: edit completion rates for people shown Tone Check decreased only slightly by -3.2% (-1.6) percentage points.
    2. Platform: the relatively small decrease in edit completion rate was concentrated on Desktop (-2.6%), with no significant change on Mobile Web.
      1. Note: the decrease in completion rate does not exceed over 10% until more than 10 tone checks are presented in a single editing session. For these edits, edit completion rate decreased to 44.3% (a -12% decrease from the control). These edits represent only 3% of edits and potentially low quality edits that we’d want to deter.
  6. Retention rate
    1. Overall: People who encountered Tone Check are 24% more likely to return again to make a constructive edit in their second week. Retention rates increased from 5.8% to 7.2% when Tone Check was shown (+1.4 percentage points).
  7. Guardrail metrics
    1. Edit completion rate: no significant decreases in edit completion rate.
    2. Revert rate: no significant decreases in revert rate.
    3. Check dismissal rates: editors declined a tone check and selected “the tone is appropriate” at 16.4% of all published edits where Tone Check was shown. This excludes edits that were reverted within 48 hours.
      1. For comparison, this is higher than the rates observed for Reference Check (6.6% of editors indicated that the content they were adding did not require a reference) and lower than Paste Check (30% of editors indicated that they wrote the content).
    4. Block rate: people are not blocked at a higher rate after being shown Tone Check.
A tree diagram showing Tone Check A/B experiment design.
Diagram showing Tone Check A/B experiment design.

But et structure de l'expérimentation

  • Wikis: French, Japanese, Portuguese
  • Timing: 3 September 2025 – 28 January 2026
  • Participants: Unregistered users and registered editors with ≤100 edits
  • Platform(s): Desktop and mobile web
  • Primary metric: Proportion of new content edits that are reverted on the grounds of WP:NPOV (and related policies)
  • Secondary metrics: Constructive edit rate
  • Guardrail metrics: Edit completion rate, edit revert rate, edit abandonment rate, Check dismissal behavior

Indicateurs clé

On 3 September 2025, an A/B experiment of Tone Check began at the French, Japanese, and Portuguese Wikipedias. What follows, is an analysis of test events logged between 8 September 2025 and 22 November 2025. This analysis was meant to enable the team to decide the following: What – if any – UX adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the feature's impact?

Note: the findings that follow are not statistically significant. We expect to be able to share statistically significant conclusions in January 2026 via T387918 .

Findings

  • Activation frequency
    • Tone Check was shown at least once in 9% of all published new-content edits by newer editors
      • Tone Check was shown in 9.5% of all published new content edits on desktop and 7.6% of all published new content edits on mobile.
  • Edit completion rates
    • Overall: edits shown Tone Check were completed at a lower rate (66.7%) than eligible edits not shown Tone Check (68.3%), a 2.3% relative decrease.
      • On mobile web there was a 7.6% relative increase for the treatment group (69.4%) compared to the control (64.5%).
      • On desktop there was a 5.2% relative decrease for the treatment group (65.8%) compared to the control (69.4%).
  • Revert rates
    • Overall: there have been no significant changes in the revert rate of all new content edits overall or by platform or Wikipedia. However, we’ve observed decreases in revert rate when limiting to edits where Tone Check was shown or eligible to be shown.
    • Platform: when we look at the revert rate of edits where Tone Check was shown at least once in an editing session compared to eligible edits in the control group, we see that:
      • On desktop, we observed a -5.3% decrease in the revert rate
      • On mobile, we observed a -19% decrease in the revert rate
    • For edits shown Tone Check and where text was revised to address the issue, we're currently seeing almost a 2x decrease in revert rate compared to eligible control edits.
  • Blocks
    • Less than 1% of users have been blocked after publishing an edit where at least one tone check was shown.
  • Model speed
    • ~0.6% of all published edits (264 edits) in the AB test were saved before the model returned an evaluation. The majority of these edits occurred in the control group and on desktop.

Next steps

The Editing Team will proceed with the Tone Check A/B experiment without making adjustments to the intervention's user experience or experiment design.

The above is grounded in the fact that:

  1. Tone Check is shown within a sufficient number of new content edits.
  2. Edits shown Tone Check are completed at a lower rate (66.7%) than eligible edits not shown Tone Check (68.3%), a 2.3% relative decrease.
    1. This slight decrease is not surprising as we are introducing an extra step in the workflow; as it's below a 10% relative difference we do not see signs of concern at this time.
  3. Published new-content edits shown Tone Check (compared to edits eligible for Tone Check to be down) are reverted less frequently.
    1. On desktop, we observed a -5.3% decrease in the revert rate
    2. On mobile, we observed a -19% decrease in the revert rate

Configurabilité

Tone Check will be implemented – like all Edit Checks – in a way that enables volunteers to explicitly configure how it behaves and who Tone Check is made available to.

Configurability happens on a per project basis so that volunteers can ensure the Tone Check experience is aligned with local policies and conventions.

The particular facets of Tone Check that will be community configurable are still being decided. If there are particular aspects of Tone Check that you think need to be configured on-wiki, we ask that you share what you are thinking in T393820 or on the talk page.

Calendrier

Year Month Activity Notes
2026 avril Tone Check deployed as default-on feature at French, Japanese, and Portuguese Wikipedias
février Tone Check A/B experiment concluded and results published
2025 juin Tone Check Model card published
Published summary of recent en.wiki conversations (on-/off-wiki)
Local (en.wiki) Tone Check project page published
Call held with en.wiki volunteers on Discord
Discussion about Tone Check emerges at en.wiki
mai Tone Check presented during the ESEAP Summit
Invitations published on volunteer talk pages seeking help with model review[4]
MassMessage sent inviting volunteers to review Tone Check Model
Published Mediawiki page inviting volunteers to sign up to review the Tone Check model.
Tone Check presented during Afrika Baraza Annual Planning Call
Announcement about volunteer-led model review published
avril Tone Check community conversation held
Invitation to Tone Check-focused community conversation published
Tone Check (then called "Peacock Check") proof of concept presented during the "CEE Catch up Annual Planning Workshop."
Tone Check (then "Peacock Check") community conversation invitations published[5][6][7][8][9]
mars Tone Check project page published
Work on Tone Check announced on mediawiki.org
2024 novembre Work on Paste Check Announced
août WMF CPTO (Selena Deckelmann) shares Reference Check demo at Wikimania
juin Link Check deployed to all wikis
mars Reference Reliability Check deployed to all wikis
2023 octobre 2023 Reference Check deployed to first Wikipedias
février 2023 Editing Teams publishes summary of early community conversations
2021 août Idea of Edit Check presented at Wikimnia

Historique

Tone Check, and the broader Edit Check initiative, is a response to a range of community conversations and initiatives. Some which include those listed below. For more historical context, please see Vérification des modifications .

Vérification des modifications

Cette initiative fait partie du projet de Vérification des modifications – un effort pour que les contributeurs fassent connaissance pendant qu’ils modifient, avec les retours exploitables sur les règles Wikipedia.

Le contrôle des modifications est conçu pour impacter simultanément deux groupes clé de personnes.

les bénévoles expérimentés qui ont besoin de :

  1. Soulagement ne réparant les dommages évitables
  2. Capacité à se confronter à la complexité

Bénévoles nouveaux et récents qui ont besoin de :

  1. Commentaire actionnable
  2. Des opportunités convaincantes de contribuer
  3. Clarifier ce que l'on attend d'eux

Questions fréquentes

Pourquoi le contrôle du ton utilise l'apprentissage machine ?

L'apprentissage automatique permet aux projets Wikipédia de détecter une langue promotionnelle ou un ton qui n'est pas neutre avant la publication.

Quel modèle d'apprentissage automatique est utilisé par le contrôle du ton ?

Le modèle BERT à source ouvert est utilisé. Le modèle utilisé n'est pas un grand modèle linguistique (LLM large language model). C'est en fait un modèle de langage plus petit, que l'équipe d'apprentissage automatique préfère, parce qu'il nous dit à quel point chacune de ses prédictions est probable, et il est plus facile à adapter à nos données personnalisées.

Quelles langues la vérification du ton prend (prendra) en charge ?

À partir de , le contrôle du ton prend en charge les langues suivantes : English, Spanish, French, Japanese, Portuguese

Pour commencer, voir T388471#10781906 pour les détails concernant la manière dont ces langues ont été priorisées.

L'objectif pour le contrôle du ton reste la prise en charge de toutes les langues. De nouvelles langues sont ajoutées régulièrement et évaluées par les membres de la communauté .


J'ai essayé la fonction mais le contrôle du ton n'est pas apparu, pourquoi ?

Si le contrôle du ton n'apparaît pas quand vous pensez qu'il le devrait, c'est probablement parce que le modèle n'estime pas que le texte que vous avez ajouté a un problème de ton.

Où être suffisamment certain dans ce contexte représente le seuil de probabilité auquel le contrôle du ton a été initialisé.

Ce seuil de probabilité peut être configuré dans chaque wiki .

Que signifie un « seuil de probabilité de 0,8 » pour la moyenne du modèle ?

Le modèle peut faire beaucoup de prédictions différentes mais il est configuré pour agir sur celles dont nous sommes parfaitement sûrs. Ou dit autrement, afficher le contrôle du ton là où il le faut est plus important que de l'afficher partout où c'est posssible.

Quelles mécanismes de trace (s'ils existent) seront implémentés sur le wiki afin que les bénévoles puissent savoir quand la vérification du ton s'est affichée ?

Pour commencer, le contrôle du ton va introduire deux nouvelles balises d'édition :

  1. editcheck-tone-shown – Cette balise sera ajoutée à toutes les modifications pour lesquelles un contrôle du ton au moins est affiché.
  2. editcheck-tone – Cette balise sera ajoutée à toute modification faite via l’Éditeur visuel par un utilisateur qui a publié moins de 100 modifications cumulées que le modèle du contrôle de ton a estimé à 80% comme contenant un problème de ton.

Cette approche suit ce qui était implémenté pour le contrôle des références.

Est-il possible de voir ce qui a été modifié par l'utilisateur après que le contrôle du ton ait été affiché ?

Cela nécessiterait de stocker à la fois le texte original et le texte modifié. Nous ne prévoyons pas de travailler sur cette question pour le moment, car elle a des implications techniques (nouvelle table dans la base de données) ainsi que des implications juridiques (actuellement, seule la version publiée est prise en compte).

Pourquoi ne pas implémenter le contrôle du ton en tant que filtre anti abus ?

Une idée clé du contrôle du ton et plus largement, du système Vérification des modifications est que les nouveaux venus agissent d'autant plus quand on leur soumet des commentaires

  1. Evoqué pendant que les nouveaux arrivants mettent à jour
  2. Affiché en relation avec le contenu spécifique en rapport avec le commentaire

Tel qu'il est implémenté actuellement, le filtre anti-abus ne peut pas couvrir les commentaires de sorte à s'aligner sur ces deux principes de conception.

Qu'allons-nous faire pour nous assurer que le contrôle du ton ne fera pas que les personnes publient des formes plus subtiles de publicités, d'allégations, ou utilisent un langage autrement abject plus difficile à détecter par le modèle et les personnes ?

  1. Journalisation et vérification : à chaque fois que le contrôle du ton s'affiche durant une session d'édition résultant en une publication, la balise masquées d'édition editcheck-tone-shown est ajoutée. Les patrouilleurs et les relecteurs peuvent à leur tour utiliser cette balise d'édition pour identifier les modifications qui pourraient nécessiter un examen plus approfondi et, par la suite, décider quelle modération, le cas échéant, ils devraient adopter en réponse. En outre, étant en mesure d'isoler séparément les modifications où le contrôle du ton s'est affiché, ensemble – le personnel et les bénévoles – nous pouvons évaluer l'impact holistique de la fonctionnalité. Cela inclut la possibilité de multiplier les formes de langage promotionnel, abject ou autrement subjectif qui sont plus difficiles à détecter pour le modèle et les personnes.
  2. Expérience limitée dans le temps : le contrôle du ton est développé avec l'idée qu'il n'est pas encore clair si l'intervention sera effective au moment de délivrer l'impact qu'il est censé causer. Pour évaluer cela, l'équiper réalisera une expérience contrôlée, avec des dates de début et de fin définitives. Les données recueillies par cette expérience permettront au personnel et aux bénévoles d'évaluer ce qu'ils devront faire ensuite.

Ensemble, nous pensons que ces deux points fourniront au personnel et aux bénévoles les données qualitatives et quantitatives nécessaires pour évaluer l'impact global de la fonctionnalité. En outre, nous pensons que la nature de l'expérience, limitée dans le temps et avec un objectif bien défini, nous permettra d'apprendre collectivement de sorte à minimiser les dommages irréparables faits aux wikis.

Quels les moyens ont les bénévoles pour agir sur le comportement du contrôle du ton et à qui cela est possible ?

Les bénévoles peuvent, sur la base du projet, configurer les aspects suivants de la vérification du ton :

  • Les états du compte (connecté, non connecté) que le contrôle de ton peut voir pendant la modification
  • nombre de modifications qu'un utilisateur doit avoir publiées pour déclencher le contrôle du ton. Par défaut, le contrôle du ton apparaît aux personnes qui ont publié moins de 100 modifications cumulées.
  • noms des sections ignorées par le contrôle du ton.

Voir Edit check/Configuration pour plus de détails. Veuillez ouvrir une discussion sur Talk: Edit check/Configuration si vous pensez qu'il y a des aspects supplémentaires du contrôle de ton que vous pensez intéressants s'ils étaient configurés sur le wiki.

Quel contrôle les bénévoles ont-ils sur le comportement du modèle ?

En plus de pouvoir configurer la manière dont le contrôle du ton se comporte , le modèle sous-jacent du contrôle du ton est entraîné sur une base continue.

De cette manière, le contrôle du ton reste synchronisé avec les façons dont les bénévoles font évoluer les politiques de Wikipédia pour adapter les changements dans la façon dont les éditeurs modifient.

See also

  • Growth/Revise Tone, a structured task which uses the same model as Tone Check to suggest edits to newcomers via a feed of articles

Références

  1. peacock check littéralement le contrôle du paon, consiste à renseigner les rédacteurs qui ajoutent du texte à un article Wikipedia que d'autres utilisateurs pourraient l'interpréter comme étant d'un ton pas neutre, promotionnel, ou autre...
  2. nouveaux bénévoles représente les personnes ayant publié 100 éditions cumulées ou moins.
  3. Emphasis on "help" seeing as how all decisions will depend on a variety of data, all of which need to be weighted and considered to make informed decisions.
  4. en:User talk: Chipmunkdavid en:User talk:NightWolf1223 en:User talk: Parksfan1955 en:User talk: The Grid en:User talk: Bunnypranav en:User talk: Xandru4 en:User talk: Meritkosy en:User talk: Fuzheado ja:User talk: Wadakuramon ja:User talk: Saebo ja:User talk: VZP10224 ja:User talk: Hexirp ja:User talk: Afaz
  5. Tech/News/2025/17
  6. en:Wikipedia talk:Manual of Style/Words to watch
  7. Discussion Projet:Aide et accueil
  8. Wikipedia talk:Growth Team features
  9. Wikipedia talk:WikiProject Editor Retention