JADE/Implementations

This page describes a set of thoughts about implementations of JADE. See T166053 for the tracking task.

System requirements

 * REST HTTP API
 * Accessible from the public Internet.
 * Allows all relevant operations (save/update judgement, suppress comment, etc.)
 * Integrates with MediaWiki authentication (OAuth2). Unprivileged operations don't require credentials, privileged ones require appropriate onwiki privileges.
 * Queryable in some basic ways (e.g. give me all judgements for an artifact)
 * Note that the API will only be able to access artifacts by their primary ID, we cannot join e.g. on page_id to get all judged revisions of that page.
 * Operations are checked for validity before we respond. If the user is disallowed, the entity doesn't exist, or there's another reason preventing this action, the HTTP response will indicate failure.
 * Operations are atomic. If the HTTP response indicates success, then the client can safely assume that the operation will eventually complete successfully.  This "eventual" caveat is because our event-passing takes some time to propagate.
 * ORES integration
 * Provides fast access to judgements to integrate with ORES predictions (ORES response time for cached scores is ~50ms. We shouldn't be slower than 100ms)
 * Some concept of overlap between ORES models/scores and JADE artifacts. Judgments if they are available should be returned along with regular score requests.
 * Public query-ability
 * Users are able to run SQL (or similarly easy query language) that joins judgements with production database tables with the wiki entities being judged.
 * Queries can be run from the public Quarry interface.
 * JADE comments can be joined and full-text searched.
 * Full and partial dumps available for internal and external use
 * WMF will maintain a full, private dump of every action JADE has recorded.
 * Dumps will be made available to the public, which have suppressed content redacted. Something will have to be done about future, retroactive suppressions.
 * Curation activities
 * Full integration with MediaWiki curation tools.
 * Comments and judgments appear in Special:RecentChanges in an obvious and usable way.
 * Comments and judgments appear in Special:Contributions.
 * Comments and judgments appear on Special:Watchlist when the judged wiki entity is being watched.
 * Blocked users cannot submit any content to wikis where they are blocked.
 * Extension:AbuseFilter integration.
 * When a user account is suppressed or renamed, their judgments are modified accordingly.
 * Users (which privileges?) will be able to suppress comments and usernames that are offensive or violate privacy.
 * We cannot add significantly to the existing review load, currently c. 350-500 suppressions per month.
 * Event-based, can easily add consumers
 * The feed of JADE actions will be available, through a stable event-based interface. Consumers will be able to catch up with any missed messages, and don't need to be constantly online to capture the entire feed.
 * Third-party consumers will need to agree to our redaction terms, so that suppressions are propagated.
 * Judgments can include quantitative scores on multiple optional "schemas", and freeform comments.
 * Schemas look like "Is this good faith?": "good faith" : "bad faith". They are versioned.
 * We support discussion threads, they are rooted at a wiki entity and may refer to specific judgments or specific scores.
 * We won't be implementing our own discussion threading, but will rely on an existing platform such as Structured Discussions.
 * It will be possible to navigate back from the discussion to the wiki entity and judgments.
 * Judgments will target wiki entities "edit" (revision as a change) and "version" (revision as snapshot of a page) separately, with different schemas available for each type.
 * A given wiki entity will have at most one "preferred" JADE judgment.
 * Editors can BOLD-Revert-Discuss to collaboratively set the "endorsed" judgment.
 * There's some open discussion at "Talk/JADE#Judgment Bold-Revert-Discuss vs. optional consensus"
 * Any UI for JADE will be fully translatable and language will be drawn from a shared store.
 * Metrics will be made available for JADE activity per-wiki, and for system monitoring.

Auditing

 * Reporting problems with ORES
 * User sees an incorrect label, responds by creating a new judgment on that artifact.
 * Grounded theory
 * Researcher makes queries using the API, makes SQL queries, or processes the raw data dump.
 * Effective refutation of ORES predictions
 * User can see other users' judgments. They are first-class in the API, and presented alongside ORES scores.
 * Showing improvements in ORES
 * Researcher uses JADE to analyze ORES's performance.

Label storage

 * Judgments can be used as our label store
 * Wiki Labels reads from and writes to the JADE store.
 * Researcher can upload lists of artifacts needing judgments.
 * Improve labels by directing attention to borderline cases
 * Researcher identifies borderline judgments.
 * Wiki Labels presents chosen artifacts for judgment.

Replicated data
We want a data store which can be efficiently queried by the ORES backend. This does't have to be the only store, or even the authoritative master. Using a messaging system like EventBus would allow us to mirror the data to everywhere that we might need it.

EventBus
Our API service would send incoming events to EventLogging and ChangePropagation. These would consist of new judgment, judgment suppression, and possibly changes to freeform comments.

Consumer requirements
We need realtime consumers to legally agree to honor suppression events. Ideally, reading is done using a standard frontend to EventLogging or EventBus.

History and suppression
We'll need to support change history and admin suppression.

ContentHandler
We could get a lot of MediaWiki UI for free, by implementing our storage as a ContentHandler. This would come with additional complications and restrictions, even as we win some machinery.

Prototype
We'll create a working API and deploy to the Cloud VPS. This release is for developers only, and can be used to test client implementations. It will not be able to produce any content visible from Wikimedia sites. See the Phabricator task for a detailed specification, and the list of out-of-scope features.

Minimum Viable Product (MVP)
This is our first release that is capable of being written to and read from Wikimedia sites.