Topic on Talk:ORES/Paper

I want to talk about past work on this and how it works for ORES.

Right now, ORES' primary mechanisms for accountability look a lot like the rest of software around Wikipedia. We have public work boards, public processes, public (machine readable) test statistics, and we publish datasets for reuse and analysis. We encourage and engage in public discussions about where the machine learning models succeed and fail to serve their intended use-cases. Users do not have direct power over the algorithms running in ORES, but they can affect them through the same processes that are infrastructures are affected in Wikipedia.

This may not sound as desirable as a fully automated accountability dream that allows users more direct control over how ORES operates, but in a way, it may be more desirable. I like to think of the space around ORES in which our users build false positive reports and conversations take place as a massive boundary object through which we're slowly coming to realize what types of control and accountability should be formalized through digital technologies and/or rules & policies.

At the moment, it seems clear that the next major project for ORES will be a means to effectively refute ORES's predictions/scorings. Through the collection of false positive reports and observations about the way that people use them, we see a key opportunity to enable users to challenge ORES' predictions and provide alternative assessments that can be included along with ORES' predictions. That means, tool developers who use ORES will find ORES' prediction and any manual assessments in the same query results. This is still future work, but it seems like something we need and we have already begun investing resources in bringing this together.

Here's some notes that JtsMN posted in the outline.

• Jake's Notes
  • accountability thread for future discussion, with an example
    • models that stop discriminatory practice against anons may have other effects
    • perhaps switching to GradientBoosting from LinearSVC helps anons, and harms e.g. the gender gap

My thinking on this line is mostly as a discussion point. I think the point you make above is reasonable, and for a given Discussion section, I think subsections of "fully automated accountability dream" and "sociotechnical oversight" are both super interesting.

Also, to be clear, I very much agree that "effective refutation" is a super interesting direction for accountability.

Agreed on the sub-sections. JtsMN, how would you define the "fully automated accountability dream"? Here's what I'd do about "sociotechnical oversight".

Sociotechnical oversight

• Thinking about boundary objects. We don't yet know what types of oversight will be necessary and how people will want to engage in it.
• So we designed open channels and employed wiki pages to let others design their means of reporting false positives and other concerns.
  • Public wiki pages, web forum, mailing list, work logs, open workboard for project management and prioritization of work.
  • We also worked with local confederates from different communities to help socialize ORES & ORES-related tools as well as to design a process that would work for their community. These confederates helped us with translations and to iterate on solutions with communities who we could otherwise not effectively work with.
• Rewards:
  • We learned that humans are pretty good at "seeing" into the black box.
  • We saw effective oversight occur in some interesting cases (anon bias, Italian "ha", etc.)
  • We saw themes emerge in how people want to engage in oversight activities and this has driven the motivation for encoding some of this process in technology -- e.g. a means to review predictions and "effectively refute" them.
  • We learned certain strategies to avoid -- e.g. sending everyone to a "central wiki" to report problems and concerns didn't really work for many communities.

I'm not 100% sure what this sort of thing looks like, but I'm gonna brain-dump, and we can go from there. I think this section has to be more speculative, and less anchored in ORES experiences thus far, but I think there points at which to tie it back.

Fully Automated Accountability At Scale?

Accountability seems to have three major factors

• Verification that the system isn't biased along certain dimensions (e.g. protected groups)
• Effect sizes
• Ability to raise new dimensions along which bias is 'claimed'/hypothesized to be occurring

As such, the question of automating accountability hinges on these factors

• There are techniques that would allow achieving the first one
  • it's apparently common in ML circles to treat models as a 'black box', and seek to predict the output along different (hypothesized to be biased) dimensions
    • This is broadly automating the community review process that occurred for anons and false-positives.
  • https://arxiv.org/abs/1611.04967
• For point 2, at what point is 'a bias that has been shown' /meaningful/?
  • There are clearly meaningful examples (anons)
  • A rule of thumb used elsewhere is a definition of 'disparate impact'
    • could this be operationalized automatically?
  • This could also be one dimension in which ORES could also support standpoints
  • How does addressing one dimension of bias affect the others (the intersectionality question)?
    • e.g. the LinearSVC model is better for anons than GradientBoosting, but may harm gender bias efforts (if anons are almost always male, does enabling better anon participation harm the efficacy of WikiProject Women in Science?)
• The third is more of an open question:
  • Should it be community driven?
  • Should there be effort to automate recognition of dimensions of bias? How do we distinguish between 'bias against swear words' (statistical signal), and 'bias against anons' (harm from statistical signal), if there is no community involvement?
• While there's clearly a tension in between full automation and community participation, the question of legibility and scale is really important - as algorithmic infrastructure is formalized, what will failing the community actually look like?
• A 'fully automated accountability system', like ORES, risks operationalizing the ideologies of the builders.
  • It's not clear that full automation can ever be achieved while supporting standpoints and meaningful community oversight
  • Lowering the barriers to accountability (e.g. proposing new dimensions of hypothesized bias, etc.) at scale may be a fundamentally sociotechnical problem
  • However, automating "due diligence" may be 'good enough automation'. This could mean:
    • dimensions of accountability (e.g. protected classes)
    • others?