|This page in a nutshell: introducting a new software initiative to provide human feedback about AI judgments.|
How will JADE be used?
Anyone interested in writing judgments or tools to create judgments should start by reading JADE/Integration. JADE data is stored as regular wiki pages, for example https://en.wikipedia.beta.wmflabs.org/wiki/JADE:Diff/376901 The format is a bit unpleasant to write by hand, and unfortunately we aren't able to offer a reference UI. We expect a small ecosystem of user interfaces to be developed over time.
We hope that JADE will become useful to the patroller community especially, as a way to coordinate work.
JADE data should also become an important reservoir of counterexamples which help challenge assumptions and mitigate biases in ORES or other AIs. Judgments in JADE can be used to audit ORES (e.g. tracking bias) as well as to retrain ORES. Doing this in an open and collaborative way will encourage democratic oversight, rather than a handful of technical staff making all the decisions about how to build ORES.
Why is JADE important?
JADE will facilitate a community of people overseeing the AI, perhaps even in "partnership" with the AI. JADE is needed so that editors can effectively challenge the AIs' automated judgments. Currently this work is done ad-hoc on wiki pages. E.g. it:Progetto:Patrolling/ORES. JADE represents basic infrastructure to better support this auditing process. The goal is put more power into the hands of the people that ORES' predictions affect.
What will Jade support?
- MediaWiki integration
- Allows users to review a judgement for a wiki entity (revisions, pages, etc.)
- Allows users to submit and edit judgments
- Public API for to tool developers and extension developers (Huggle, RC Filters, etc.)
- ORES integration
- Judgments returned along with predictions.
- Consensus patterns
- Users file dissenting judgments
- Structured discussions (or talk pages) for every wiki entity
- Collaborative analysis
- Judgments open licensed and publicly accessible
- Machine readable dumps/api for generating fitness and bias trend reports
- Curation and suppression
- Recent judgments appear in Special:RecentChanges
- Basic suppression actions supported (hide comment, user, etc.)
Sign up to be contacted about discussions: JADE contact list
- "Best practices for AI in the social spaces: Integrated refutations"
- Technical work: task T148700 and its subtasks.
- JADE/Open questions
- Past examples of (manual, wiki-based) ORES auditing: