Topic on Talk:ORES review tool

I noticed the red "r"s on my watchlist and read up about ORES. I would like to assist and review the edits flagged. Can you confirm how I can review the edits? Thanks.

Hi Adamiow. We have independent efforts for *using* ORES to do reviewing work and *training* ORES to make better predictions.

Using ORES

If you are looking to help fight vandalism and revert other types of damage, I recommend reviewing the edits flagged with an "r" and using "undo" as necessary to remove those that are in fact damaging edits. ORES recommends edits for review, but ultimately, human judgement is required for determining if an edit needs to be undone.

Training ORES

If you'd like to help us make ORES more accurate, check out en:WP:Labels and en:WP:Labels/Edit_quality. See m:Wiki labels for more information about how the system works. You can use the system an its associated gadget to help us label edits.

Thanks Halfak (WMF). It would seem useful to me to have the ability to mark an edit as ok (at least during the initial phase) to help improve the accuracy.

Copy-pasting from a post that I made on the AI mailing list a while back:

So, in order to avoid a bias feedback loop, we don't want to feed any observations you made *using* ORES back into the model -- since ORES' prediction itself could bias your assessment and we'd re-perpetuate that bias. Still, we can use these misclassification reports to direct our attention to problematic behaviors in the model. We use the Wiki Labels system[1] to gather reviews of random samples of edits from Wikipedians in order to train the model.

Misclassification reports

See https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Misclassifications/Edit_quality

We're still working out the Right(TM) way to report false positives. Right now, we ask that you do so on-wiki and in the future, we'll be exploring a nicer interface so that you can report them while using the tool. We review these misclassification reports manually to focus our work on the models and to report progress made. This data is never directly used in training the machine learning models due to issues around bias.

Wiki labels campaigns

In order to avoid the biases in who gets reviewed and why, we generate random samples of edits for review using our Wiki Labels[1] system. We've completed a labeling campaign for English Wikipedia[2], but we could run an additional campaign to gather more data. I'll get that set up and respond to this message when it is ready.

1. https://meta.wikimedia.org/wiki/Wiki_labels
2. https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality