2025 RecentChanges Language Agnostic Revert Risk Filtering

Start:	2025-04-01
End:	2025-06-30
Team members:	Moderator Tools, Machine Learning

Translate this page

Background

Recent changes is a feed of all edits on a wiki. Patrollers and moderators rely on Recent changes to identify edits requiring action.

The ORES extension fetches and stores Machine learning model scores for recent edits, which are used to filters changes on Special:RecentChanges, highlighting the ones with a high risk of being reverted. These filters help patrollers to find edits that are more likely to be damageable for the wiki.

The ORES infrastructure is being deprecated by the Machine Learning team. Plus, this extension and the associated filters are only enabled on select wikis.

The Moderator Tools team would like to add a similar filter to all Wikipedias, by using the Language-Agnostic Revert Risk.

The Moderator Tools team has determined a list of wikis would benefit from this filter (and do not have high technical hurdles), regardless of whether or not the ORES extension is currently enabled for that wiki.

What is changing

A new filter is added to Recent changes filters:

Likely to be reverted: Detects revisions that might be reverted, but does not assess the user's intention.

This filter is shown under a new section: "Risk of being reverted".

It can be used like any other filter.

This filter identifies edits that are likely to be reverted by scoring an edit against a revert risk model threshold. It is a tool to assist Recent changes reviewers. However, some of the changes filtered won't need to be removed, while some other changes that should be reverted might not be present in the filtered results.

In the future, this new filter could replace existing quality filters, powered by ORES, which are deployed at a few wikis.

Likely to be reverted threshold analysis findings

Overview

The filter works by scoring an edit against a revert risk model and if the score is above the specified threshold, then the edit is categorized as "likely to be reverted".

As part of the threshold analysis, we wanted to better understand the distribution of revert risk scores and how they corresponded to real-world activities (i.e. was the edit actually reverted) so that we can provide a process to arrive at buckets for the scores, e.g.“Likely to be reverted”.

Methodology

We reviewed a sample of article namespace edits completed on Indonesian Wikipedia. The edit sample included if the edit was reverted within the reviewed timeframe and the revert risk score based on the scores provided by the language-agnostic model.

Some data analysis was performed to target a threshold which resulted in an approximately 15% false positive rate. In other words, approximately 15% of edits or approximately 1 out of 7 edits shown as likely to be reverted are falsely classified as such. We targeted a 15% false positive rate as it seemed like an appropriate number of false positives for a user-facing feature.

The threshold analysis is repeatable (using this notebook). In order to target different levels of false positive rates (lower or higher than 15%) you can re-run the notebook with different parameters and arrive at a more optimized threshold. The threshold can then be updated via configuration like in this patch.

Note, this process is only available for language agnostic wikis currently.

Thresholds

As a result of the analysis we were able to arrive at the following threshold for Indonesian Wikipedia.


Revert Risk Score	Meaning	Likely Outcome/Tradeoffs	Supplemental data
Over 0.85	Very likely for edit to be reverted.	Moderate false positive rate with high false negative rate. Correctly identifies a little over half (59.4%) of edits that were actually reverted. This threshold is good at catching the majority of edits that should be reverted (high recall), but it also flags some edits (~15%) as should be reverted when they weren't (low precision).	Recall: 0.594 Precision: 0.239 False Positive Rate: 14.5% False Negative Rate: 40.6%


Wiki	Optimal threshold
cywiki	0.11076631
simplewiki	0.90655595
bewiki	0.5724957
kkwiki	0.6048708
nnwiki	0.41624594
mkwiki	0.5058307
lawiki	0.6340628
afwiki	0.7369383
tewiki	0.36725548
mrwiki	0.86735326
swwiki	0.74820924
mlwiki	0.93626505
iswiki	0.88926035
pawiki	0.545889
hawiki	0.4823182
tlwiki	0.60741657
bnwiki	0.64658594
trwiki	0.6082413
azwiki	0.5366108

FAQ

Sometimes, the filters show edits that are not vandalism. Is it normal?: Yes, some edits are false positives, about 15% of the diff filtered. False positives are part of the detection. We can reduce this percentage, but then the number of edits being detected as vandalism would drastically decrease.

How to improve the false positive rate?: To improve the false positive rate, you can increase the threshold to a higher number as desired.

Our language has a high threshold, meaning that not many vandalisms are detected, right?: We are stricter on classifying something as vandalism, but at the same time we are confident that our decision is reliable in not making false predictions.

Targeted wikis

The changes will be applied to wikis identified in T392144:

Indonesian Wikipedia (pilot wiki)
Welsh Wikipedia
Simple English Wikipedia
Belarusian Wikipedia
Kazakh Wikipedia
Norwegian Nynorsk Wikipedia
Macedonian Wikipedia
Latin Wikipedia
Afrikaans Wikipedia
Telugu Wikipedia
Marathi Wikipedia
Swahili Wikipedia
Malayalam Wikipedia
Icelandic Wikipedia
Punjabi Wikipedia
Hausa Wikipedia
Tagalog Wikipedia
Bengali Wikipedia
Turkish Wikipedia
Azerbaijani Wikipedia

Key Milestones

Identify target wikis T392144
Deploy only to idwiki initially to validate process
Deploy to remaining wikis on list
- Including: Deploy to two wikis that have more complexities phab:T395668

Help & Learning More

The interface can be translated at translatewiki.

We welcome your questions and comments at the talk page.