Moderator Tools/Automoderator/Testing

The Moderator Tools team is building Automoderator - a tool which can automatically revert bad edits based on a machine learning model. To help communities test and evaluate Automoderator's accuracy, we are making a test spreadsheet available with data on past edits and whether Automoderator would have reverted them or not.

The decisions Automoderator would make are a combination of a machine learning model score and internal configuration. The model will be re-trained and improve over time, but we also want to understand what internal configuration rules we can apply to improve Automoderator's accuracy. For example, we have found that Automoderator often mis-judges users reverting their own edits as vandalism, so we could avoid taking action on these edits. We would like to find other examples of these kinds of edits, and need your help to do so.

How to test Automoderator

 * If you have a Google account:
 * Make a copy of this spreadsheet by clicking File > Make a Copy ...
 * After your copy has loaded, click Share in the top corner, then give any access to swalton@undefinedwikimedia.org, so that we can aggregate your responses to collect data on Automoderator's accuracy.
 * Alternatively, you can change 'General access' to 'Anyone with the link' and share a link with us directly on-wiki.
 * Alternatively, use this link to download the file.
 * After adding your decisions, please send the sheet back to us at swalton@undefinedwikimedia.org, so that we can aggregate your responses to collect data on Automoderator's accuracy.

After accessing the spreadsheet...


 * 1) Follow the instructions in the sheet to select a random dataset, review 30 edits, and then uncover what decisions Automoderator would make for each edit.
 * 2) Feel free to explore the full data in the 'Edit data & scores' tab.
 * 3) If you want to review another dataset please make a new copy of the sheet to avoid conflicting data.
 * 4) Join the discussion on the talk page.

Details
Automoderator's model is only trained on main namespace Wikipedia pages, so the dataset is limited to Wikipedia article edits. Further details can be found below:

Internal configuration
In the current version of the spreadsheet, in addition to considering the model score, Automoderator does not take actions on:


 * Edits made by administrators
 * Edits made by bots
 * Edits which are self-reverts
 * New page creations

The datasets and list above will be updated as testing progresses if we add new exclusions or configurations.

Caution levels
In this test Automoderator has five 'caution' levels, defining the revert likelihood threshold above which Automoderator will revert an edit.


 * At high caution, Automoderator will need to be very confident to revert an edit. This means it will revert fewer edits overall, but do so with a higher accuracy.


 * At low caution, Automoderator will be less strict about its confidence level. It will revert more edits, but be less accurate.

The caution levels in this test have been set by the Moderator Tools team based on our observations of the models accuracy and coverage. To illustrate the number of reverts expected at different caution levels see below:

TODO - https://phabricator.wikimedia.org/T348869

Score an individual edit
If you want to get a Revert Risk score for an individual edit, you can do so with the LiftWing API ... TODO

https://en.wikipedia.org/wiki/User:Samwalton9_(WMF)/revertrisk.js

Note that this is just the model score, and does not take into account Automoderator's internal configurations as detailed above.

Further details