ORES/New model checklist
- 1 Overview
- 2 Step 1: Determine which models need to be built
- 3 Step 2: Create tasks in Phabricator
- 4 Step 3: Compile the test dataset.
- 5 Step 4: Create badwords and informal words lists
- 6 Step 5: Add the new model to configuration files and Makefiles
- 7 Step 6: Train the new model
- 8 Step 7: Test, cross-validate, review model health
- 9 Step 8: Add and commit the model
- 10 Step 9: Deploy the new model
- 11 How to deploy server and MediaWiki components of ORES on a new wiki
- 12 See Also
This is a guide to training new ORES models and enabling them on wikis.
This guide is intended for members of the Scoring Platform team.
See historical examples of adding new models: task T130213
Step 1: Determine which models need to be built
Each model can target one set of classifications. The current norm is to begin with
goodfaith models for a single language wiki database.
Note: Obtaining labeled training observations can require a large investment of volunteer time. This can be a limiting factor when making decisions about which models to build.
Step 2: Create tasks in Phabricator
Create the following set of tasks in for Scoring Platform in Phabricator
- Optional parent epic
- Task to gather labeled test data.
- Task to engineer language features
- Task to collect word lists.
- Task to train, test, and tune the model (blocked by test data).
- Task to deploy the model (blocked by training).
Step 3: Compile the test dataset.
- For edit quality, run a WikiLabels campaign on the target wiki.
- File a Phabricator ticket to set up the campaign.
- Announce the campaign on-wiki.
- Run the campaign until it is complete. Update the community regularly during the campaign. Typically, around 5,000 articles will need to be assessed.
- For an article quality model, we'll need to collect the set of all articles already given evaluations, grouped by quality classification.
- For some models, test data may be extracted by using database queries.
Step 4: Create badwords and informal words lists
Badword lists (AKA BDWS lists) and informal words lists have already been generated for a number of languages. An appropriate BDWS list for the new model may be found in the existing revision scoring word lists.
Lean more about how to sort BDWDS-generated words lists.
Step 5: Add the new model to configuration files and Makefiles
- For example, https://github.com/wiki-ai/editquality/pull/68/commits/4a9916e4d0291db39ca9d916600ecd39f1c1ec15
- TBD: and more?
Step 6: Train the new model
(p3)ladsgroup@ores-compute-01:~/editquality$ make models/frwiki.damaging.gradient_boosting.model
Step 7: Test, cross-validate, review model health
Step 8: Add and commit the model
If the model requires the installation of a new language dictionary, add the dictionary to the ORES base config in puppet.
Step 9: Deploy the new model
The final step is to deploy the new model.
There are a number of spaces where new models can be deployed. Refer to the following chart to determine the appropriate space.
|Suggested use||Serving real-time requests||Making sure we don't break production||Experimenting with new code and running analyses||Making sure we don't break the experimental install|
|Stability||Stable||Not stable||Mostly Stable||Not stable|
|Performance||Fast||Extremely limited||Pretty fast||Extremely limited|
|Code version||Stable code||Experimental code||Experimental code||Experimental code|
|Parallel Requests||Up to 2 parallel requests per second||Up to 1 request per second||Up to 4 parallel requests per second||Up to 1 request per second|
How to deploy server and MediaWiki components of ORES on a new wiki
- Follow all of the steps on the beta cluster.
$wmgUseORESfor the wiki.
- Configure wiki thresholds to reasonable defaults (how do we guess these?) in
- Request new DB tables.
For additional information contact the Scoring Platform team.