Wikimedia Scoring Platform team/FY2019



The Scoring Platform team is an experimental, research focused, community supported, AI-as-a-service team. Our work focuses on balancing the efficiency that machine classification strategies bring to wiki-processes with transparency, ethics, and fairness. Our primary platform is ORES, an AI service that supports wiki processes like vandal fighting, gap detection, and new page patrolling. The current set of ORES-supported products are loved by our communities and our team's work, supporting overloaded community processes with AI, has shown great potential to enable conversations about growing our community (Knowledge Equity). In this proposal, we'll describe what we think we can accomplish given our current, minimal staffing. We'll also propose to fully staff the team along the lines of the original FY2018 Scoring Platform proposal so that we can expand our in the critical area of bias detection and mitigation.

Overview of FY2018
Last year, we invested in the Scoring Platform team by giving Aaron a budget by staffing the team with Adam Wight as a senior engineer(80%) and Amir Sarabadani as a junior engineer(50%). We also retained a contracting budget to hire experts to develop new AIs and evaluation strategies.

Despite this minimal staffing, the team has been quite successful.
 * Lots more models delivered to lots more wikis (targeting emerging communities, increasing capacity for knowledge equity)
 * Deployed ORES on a dedicated cluster and refactored the ORES extension (more uptime, evolving infrastructure)
 * Collaborated with commtech on a study of new page review issues -- trained and tested a critical technology for mitigating the issue (evolving our infrastructure and experimenting with new strategies for supporting newcomers)
 * Published papers about why people cite what they cite and the dynamics of bot governence (increasing our understanding of wiki processes)
 * We performed a community consultation and system design process for JADE, our proposed auditing support infrastructure

Current staffing
In the next fiscal year, we'd like to continue our work towards making ORES more robust and expanding our prediction models to new wiki processes and under-served wiki communities.
 * Increase model support to more wikis -- targeting emerging communities
 * Experiment with the new article routing models and expand them to more communities
 * Publish datasets and papers about the process and machine-based process augmentation

Fully-staffed
While the Scoring Platform team has been able collaborate effectively with volunteers in order to supplement it's minimal resourcing, the fact is that the development of ORES (useful AIs) and JADE (our auditing system) has been slowed substantially. Further, our bus factor is still far too low. Were we to temporarily lose the one full-time engineer on the team, development and deployments would nearly come to a halt. Or worse, if Aaron were to be lost, the majority of the team's infrastructure would leave with him. We can bring the team up to a higher level capacity and robustness by (1) promoting Amir to a full-time req holder and (2) hiring an engineering manager/tech lead to remove that burden from Aaron. This is in-line with our original plan for FY2018.

Currently, we have a large backlog of wikis that want ORES support and a we're looking at a mountain of work to bring our new auditing system, JADE, online. If ORES was developed faster, we'd have a more robust service, we'd support more wikis, and we'd develop new prediction models more quickly (Knowledge as a Service). Many of these models are targeted intended to provide fertile ground for experimentation around mixing efficient quality control with better newcomer support (Knowledge Equity). If JADE was developed faster, we'd be able to start tracking algorithmic -- the kind of problems that keep some potential contributors out -- bias much more effectively (Knowledge Equity). If the team is better staffed, Aaron would be less of a bottleneck for the team, he would be able to participate in thought leadership/outreach more effectively, and he would be able to take his vacation time.