JADE/Implementations

This page describes a set of thoughts about implementations of JADE. See T166053 for the tracking task.

System requirements

 * REST HTTP API
 * Accessible from the public Internet
 * Allows all relevant operations (save/update judgement, suppress comment, etc.)
 * Integrates with MediaWiki authentication (OAuth2)
 * Queryable in some basic ways (e.g. give me all judgements for an artifact)
 * Note that the API will only be able to access artifacts by their primary ID, we cannot join e.g. on page_id to get all judged revisions of that page.
 * ORES integration
 * Provides fast access to judgements to integrate with ORES predictions (ORES response time for cached scores is ~50ms. We shouldn't be slower than 100ms)
 * Some concept of overlap between ORES models/scores and JADE artifacts. Judgments if they are available should be returned along with regular score requests.
 * Public query-ability
 * Users are able to run SQL (or similarly easy query language) that joins judgements with production database tables with the wiki entities being judged.
 * Queries can be run from the public Quarry interface.
 * JADE comments can be joined and full-text searched.
 * Curation activities
 * Full integration with MediaWiki curation tools.
 * Comments and judgments appear in Special:RecentChanges in an obvious and usable way.
 * Comments and judgments appear in Special:Contributions.
 * Comments and judgments appear on Special:Watchlist when the judged wiki entity is being watched.
 * Blocked users cannot submit any content to wikis where they are blocked.
 * Extension:AbuseFilter integration.
 * When a user account is suppressed or renamed, their judgments are modified accordingly.
 * Users (which privileges?) will be able to suppress comments and usernames that are offensive or violate privacy.
 * We cannot add significantly to the existing review load, currently c. 350-500 suppressions per month.

Auditing

 * Reporting problems with ORES
 * User sees an incorrect label, responds by creating a new judgment on that artifact.
 * Grounded theory
 * Researcher makes queries using the API, makes SQL queries, or processes the raw data dump.
 * Effective refutation of ORES predictions
 * User can see other users' judgments. They are first-class in the API, and presented alongside ORES scores.
 * Showing improvements in ORES
 * Researcher uses JADE to analyze ORES's performance.

Label storage

 * Judgments can be used as our label store
 * Wiki Labels reads from and writes to the JADE store.
 * Researcher can upload lists of artifacts needing judgments.
 * Improve labels by directing attention to borderline cases
 * Researcher identifies borderline judgments.
 * Wiki Labels presents chosen artifacts for judgment.

Replicated data
We want a data store which can be efficiently queried by the ORES backend. This does't have to be the only store, or even the authoritative master. Using a messaging system like EventBus would allow us to mirror the data to everywhere that we might need it.

EventBus
Our API service would send incoming events to EventLogging and ChangePropagation. These would consist of new judgment, judgment suppression, and possibly changes to freeform comments.

Consumer requirements
We need realtime consumers to legally agree to honor suppression events. Ideally, reading is done using a standard frontend to EventLogging or EventBus.

History and suppression
We'll need to support change history and admin suppression.

ContentHandler
We could get a lot of MediaWiki UI for free, by implementing our storage as a ContentHandler. This would come with additional complications and restrictions, even as we win some machinery.