ORES/RCFilters

Very rough notes for now, just a transcription of what I scribbled on my notepad

To document:


 * Vocabulary (model, outcome, precision, recall, threshold, level)
 * How this appears in the UI (+Special:ORESModels)
 * Threshold selection
 * Config structure
 * Defaults (goodfaith verylikelybad missing)
 * Variability when models change
 * Disabling/skipping levels
 * Need to redo thresholds for a bunch of older models (which ones are up to date?)
 * Maintenance scripts to run during deployment
 * How to disable models (mostly bad goodfaith)
 * Thresholds in API are 1-ed for false

Threshold selection


 * maybebad: P>=0.15 or R>=0.9, tighter of the two
 * verylikelybad: P>=0.9, or R>=0.1, wider of the two (but not below P=0.6)
 * likelybad: 0.3 precision points down from verylikelybad (=P(verylikely)-0.3)? Wider of P>=0.6 and R>=0.2? Or R(verylikely)+0.1? But at least P=0.5
 * Never use thresholds with R=1 or P=1
 * likelygood: P >= 0.995 but R <= 0.9? Minimum recall for if high precision not available? No overlap with likelybad (but overlap with maybebad is OK)