Program Goals and Status for FY18/19

TEC5: Scoring Platform (ORES)

Goal Owner: Aaron Halfaker
Program Goals for FY18/19: Improve the efficiency of wiki processes and mitigate the effects of algorithmic biases that are introduced.
Annual Plan: TEC5 Scoring Platform
- Primary Goal is Knowledge Equity: Grow new contributors and content
- Tech Goal: Supporting our Community of contributors

Q1 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE team

Goal(s)

Keep ORES online and improve robustness

Status

Note: July 30, 2018

In progress

Note: August 9, 2018

In progress

Note: September 13, 2018

In progress...and will be forever and ever :)

Outcome 2 / Output 3

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. in the ORES manual

Dependencies on: Cloud Services team

Goal(s)

Documentation -- Threshold optimizations in The ORES Manual

Status

Note: July 30, 2018

In progress

Note: August 9, 2018

In progress

Note: September 13, 2018

In progress and is in review with Sarah R

Outcome 2 / Output 4

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Workshops run, papers published, datasets published, tutorials published, hackathons co-organized

Goal(s)

Developing a focus group for JADE

Status

Note: July 30, 2018

In progress

Note: August 9, 2018

In progress

Note: September 13, 2018

In progress

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

JADE --> Production

Status

Note: July 30, 2018

Blocked The block on our JADE --> Production goal is a serious threat at this point, but there's nothing to do except to follow an RFC in the short term.

Note: August 9, 2018

Blocked DBA's have confirmed a ceiling of growth for JADE in MediaWiki. An RFC discussion is scheduled for Aug. 22nd

Note: September 13, 2018

Currently still

Blocked because TechCom has requested iteration on our secondary tables for MediaWiki before they will consider closing the discussion

Q2 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

Upgrade Celery to version 4 Done
Fix logging (redirect useful logs to logstash) Done
Implement edit quality models for translatewiki In progress should be done by end of quarter

Status

Note: October 18, 2018

To do

Note: December 13, 2018

These goals are

Partially done at this point and will be fully

Done by end of quarter.

Outcome 2 / Output 3

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. on the Wikimedia blog in ORES manual

Dependencies on: Cloud Services Team

Goal(s)

Documentation -- Feature Injection in The ORES Manual In progress
Blog announcement of the Undisclosed Paid Editors dataset In progress
Resubmit ORES paper to the Journal of Social Computing N Cancelled

Status

Note: October 18, 2018

To do

Note: December 13, 2018

This is

In progress and will be finished by end of this quarter; although we will not resubmit the paper.

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

JADE --> Production (4 small wiki deployments pending local consensus)

Status

Note: October 18, 2018

In progress

Note: December 13, 2018

This is

Partially done and will be finished in early Q3

Q3 Goals

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Goal(s)

Implement basic Jade integration (Phab:T183381) (carryover + additional commitment)

Status

Note: January 10, 2019

Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

Note: February 14, 2019

Working with design on the production of wireframes/mocks In progress

Note: March 18, 2019

Stalled due to a loss in Engineering Capacity N Not done

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

Prepare ORES for kubernetes (ETA: 2 quarters)
Implement high-availability strategy for redis
Implement safe task serialization for celery (JSON)
Implement new models in response to demand
Perform research in response to community/WMF needs

Status

Note: January 10, 2019

Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

Note: February 14, 2019

New models for Italian, Korean, and German Wikipedia as well as Spanish Wikibooks.
Nearly finished with conversion from Pickle to JSON in Celery
The Teahouse AI-Hostbot study has started
Kubernetes work is In progress

Note: March 18, 2019

Large scale deployments of February achievements (new/improved models, JSON for celery, etc.)
Kubernetes work continues (as planned, this will take multiple quarters)
- Wikilabels CI is now running fully on docker

Strategy/Annual planning

Engage with 3-5 year strategy and annual planning process. Dependencies on: Audiences, Research, Search

Goals

Develop proposal for building "topic spaces" infrastructure
Develop annual plan program proposal for FY20

Status

Note: January 10, 2019

Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

Note: February 14, 2019

Topic spaces proposal has been socialized and a document is In progress

Note: March 18, 2019

Topic spaces proposal is complete and has been socialized. Talks with Audiences started Done
Annual plan work in progress In progress

Q4 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE

Goal(s)

Prepare ORES for kubernetes (ETA: 2 quarters)
Implement high-availability strategy for redis
Improve monitoring of ORES components in grafana
Implement new models in response to demand
Perform research in response to community/WMF needs

Status

Note: May 9, 2019

ORES for kubernetes is N Stalled for this quarter
Implement high-availability strategy for redis is currently N Blocked on figuring out the correct solution to take
Improve monitoring of ORES components in grafana is Done
Implement new models is In progress and we've completed svwiki article quality (model built -- pending deploy Partially done) already; srwiki goodfaith (model improved) is Done; eswikiversity (modeling In progress); dewiki model is Done but also N Blocked waiting on Audiences / Growth team for RCFilters deployment.
Perform research in response to community/WMF needs - srwiki, ruwiki, hewiki now have low quality "goodfaith" models Partially done for now. Additional work will still be In progress this quarter.
Also In progress is the work written up here.

Note: June 20, 2019

ORES for kubernetes is N Stalled for this quarter
Implement high-availability strategy for redis is currently N Blocked on figuring out the correct solution to take
Implement new models is In progress
- eswikiversity Done
- jawiki Done
- zhwiki Done
- nlwiki improvements Done
Perform research in response to community/WMF needs
- m:Research:Exploring systematic bias in ORES is still Partially done (continue into next quarter)

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: Design/Audiences

Goal(s)

Complete Jade design work. Prepare for implementation once Engineering capacity is restored.

Status

Note: May 9, 2019

This design work is In progress but progressing very slowly with reviews of the wireframes.

Note: June 20, 2019

This design work is N Postponed

Misc Goals

Hiring a Senior SWE and possibly an Engineering Manager

Interviewing is In progress for the Sr SWE and Eng Mgr roles.
Done Hired ACraze

Annual planning work: Propose growth model for ORES and related AI technology teams

This is now Done in regards to annual planning.