Wikimedia Technology/Annual Plans/FY2019/TEC5: Scoring Platform/Goals

=Program Goals and Status for FY18/19=

 TEC5: Scoring Platform (ORES)
 * Goal Owner: Aaron Halfaker
 * Program Goals for FY18/19: Improve the efficiency of wiki processes and mitigate the effects of algorithmic biases that are introduced.
 * Annual Plan: TEC5 Scoring Platform
 * Primary Goal is Knowledge Equity: Grow new contributors and content
 * Tech Goal: Supporting our Community of contributors



 = Q1 Goals =

Outcome 1 / Output 1
More wiki communities benefit from semi-automated curation support
 * ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE team

Goal(s)

 * Keep ORES online and improve robustness

Status
July 30, 2018

August 9, 2018

September 13, 2018
 * ...and will be forever and ever :)

Outcome 2 / Output 3
Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
 * Published posts about ORES, AI, wiki processes, etc. in the ORES manual

Dependencies on: Cloud Services team

Goal(s)

 * Documentation -- Threshold optimizations in The ORES Manual

Status
July 30, 2018

August 9, 2018

September 13, 2018
 * and is in review with Sarah R

Outcome 2 / Output 4
Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
 * Workshops run, papers published, datasets published, tutorials published, hackathons co-organized

Goal(s)

 * Developing a focus group for JADE

Status
July 30, 2018

August 9, 2018

September 13, 2018

Outcome 3 / Output 5
Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
 * JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

 * JADE --> Production

Status
July 30, 2018
 * ❌ The block on our JADE --> Production goal is a serious threat at this point, but there's nothing to do except to follow an RFC in the short term.

August 9, 2018
 * ❌ DBA's have confirmed a ceiling of growth for JADE in MediaWiki. An RFC discussion is scheduled for Aug. 22nd

September 13, 2018
 * Currently still ❌ because TechCom has requested iteration on our secondary tables for MediaWiki before they will consider closing the discussion

 =Q2 Goals =

Outcome 1 / Output 1
More wiki communities benefit from semi-automated curation support
 * ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

 * Upgrade Celery to version 4 ￼✅
 * Fix logging (redirect useful logs to logstash) ￼✅
 * Implement edit quality models for translatewiki ￼  should be done by end of quarter

Status
October 18, 2018

December 13, 2018
 * These goals are at this point and will be fully ✅ by end of quarter.

Outcome 2 / Output 3
Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
 * Published posts about ORES, AI, wiki processes, etc. on the Wikimedia blog in ORES manual

Dependencies on: Cloud Services Team

Goal(s)

 * Documentation -- Feature Injection in The ORES Manual
 * Blog announcement of the Undisclosed Paid Editors dataset
 * Resubmit ORES paper to the Journal of Social Computing ❌

Status
October 18, 2018

December 13, 2018
 * This is and will be finished by end of this quarter; although we will not resubmit the paper.

Outcome 3 / Output 5
Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
 * JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

 * JADE --> Production (4 small wiki deployments pending local consensus)

Status
October 18, 2018

December 13, 2018
 * This is and will be finished in early Q3

 =Q3 Goals =

Outcome 3 / Output 5
Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
 * JADE (our auditing system) accepts and stores human judgements

Goal(s)

 * Implement basic Jade integration (T183381) (carryover + additional commitment)

Status
January 10, 2019
 * Discussed that as we've just gotten back from our vacations, this work is ramping up and is

February 14, 2019
 * Working with design on the production of wireframes/mocks

March 18, 2019
 * Stalled due to a loss in Engineering Capacity ❌

Outcome 1 / Output 1
More wiki communities benefit from semi-automated curation support
 * ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

 * Prepare ORES for kubernetes (ETA: 2 quarters)
 * Implement high-availability strategy for redis
 * Implement safe task serialization for celery (JSON)
 * Implement new models in response to demand
 * Perform research in response to community/WMF needs

Status
January 10, 2019
 * Discussed that as we've just gotten back from our vacations, this work is ramping up and is

February 14, 2019
 * New models for Italian, Korean, and German Wikipedia as well as Spanish Wikibooks.
 * Nearly finished with conversion from Pickle to JSON in Celery
 * The Teahouse AI-Hostbot study has started
 * Kubernetes work is

March 18, 2019
 * Large scale deployments of February achievements (new/improved models, JSON for celery, etc.)
 * Kubernetes work continues (as planned, this will take multiple quarters)
 * Wikilabels CI is now running fully on docker

Strategy/Annual planning
Engage with 3-5 year strategy and annual planning process. Dependencies on: Audiences, Research, Search

Goals

 * Develop proposal for building "topic spaces" infrastructure
 * Develop annual plan program proposal for FY20

Status
January 10, 2019
 * Discussed that as we've just gotten back from our vacations, this work is ramping up and is

February 14, 2019
 * Topic spaces proposal has been socialized and a document is

March 18, 2019
 * Topic spaces proposal is complete and has been socialized. Talks with Audiences started ✅
 * Annual plan work in progress



=Q4 Goals =

Outcome 1 / Output 1
More wiki communities benefit from semi-automated curation support
 * ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE

Goal(s)

 * Prepare ORES for kubernetes (ETA: 2 quarters)
 * Implement high-availability strategy for redis
 * Improve monitoring of ORES components in grafana
 * Implement new models in response to demand
 * Perform research in response to community/WMF needs

Status
May 9, 2019
 * ORES for kubernetes is ❌ for this quarter
 * Implement high-availability strategy for redis is currently ❌ on figuring out the correct solution to take
 * Improve monitoring of ORES components in grafana is ✅
 * Implement new models is and we've completed svwiki article quality (model built -- pending deploy ) already; srwiki goodfaith (model improved) is ✅; eswikiversity (modeling ); dewiki model is ✅ but also ❌ waiting on Audiences / Growth team for RCFilters deployment.
 * Perform research in response to community/WMF needs - srwiki, ruwiki, hewiki now have low quality "goodfaith" models for now. Additional work will still be  this quarter.
 * Also is the work written up here.

June 20, 2019
 * ORES for kubernetes is ❌ for this quarter
 * Implement high-availability strategy for redis is currently ❌ on figuring out the correct solution to take
 * Implement new models is
 * eswikiversity ✅
 * jawiki ✅
 * zhwiki ✅
 * nlwiki improvements ✅
 * Perform research in response to community/WMF needs
 * m:Research:Exploring systematic bias in ORES is still (continue into next quarter)

Outcome 3 / Output 5
Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
 * JADE (our auditing system) accepts and stores human judgements

Dependencies on: Design/Audiences

Goal(s)

 * Complete Jade design work. Prepare for implementation once Engineering capacity is restored.

Status
May 9, 2019
 * This design work is but progressing very slowly with reviews of the wireframes.

June 20, 2019
 * This design work is ❌

Misc Goals

 * Hiring a Senior SWE and possibly an Engineering Manager
 * Interviewing is for the Sr SWE and Eng Mgr roles.
 * ✅ Hired ACraze


 * Annual planning work: Propose growth model for ORES and related AI technology teams
 * This is now ✅ in regards to annual planning.