Howdy folks. 1) Was just wondering how ORES is trained? For example, the "Vandalism" tags in PageTriage's Special:NewPagesFeed. Is there software somewhere where volunteers are presented with various pages and click a "vandalism yes/no?" button? If so where is the software? I'd like to check it out. 2) What are the PageTriage ORES configuration settings such as false positive target rate? I assume that this is a setting that can be adjusted up/down, which I assume is how anti-vandalism bot ClueBot NG achieves such a good false positive rate. I assume it's a tradeoff between false positives, and letting stuff slip through the cracks. 3) Any other hints about how ORES works in relation to PageTriage? I'll probably write some documentation about it. Thanks.
Topic on Talk:Machine Learning
I believe this is where edits are labeled, per wiki and on some quite old selection of edits. IDK if some other training options are in place. Would love to learn too!
Hey @Novem Linguae and @Ponor, me and some researchers are currently working on a project where we build a system that facilitates curating up-to-date data for training and evaluating ML models used in Wikipedia, including but not limited to ORES. We plan to recruit a small group of people for pilot testing around June. Please let me know if you're interested in participating or learning more about the project. Thanks!
Hi @Novem Linguae, there is some general information here. The models are sometimes trained using human curated training data (like @Ponor mentioned). Other times, data such as whether an edit was reverted is used. The models just output probability, the thresholds are hardcoded in the mediawiki extension itself.
Additionally, we are planning on deprecating the current ORES/Revscoring models in favor of more modern models such as RevertRisk and Outlink Topic Model which cover multiple languages and take advantage of tool such as BERT. The ORES models will still be available for legacy reasons but we won't be updating them.