Article feedback/Extended review

Because of the lack of a standard, readily-available tool to create and store quality reviews of Wikipedia content, several groups and organizations have created their own ad-hoc tools for this purpose. This page describes a standard system for such organizations to conduct open quality review of Wikipedia content, and surface the results on the article page.

This system is primarily intended for Wikipedia, but could also be used on other Wikimedia projects.

System

 * What form does the system take?

A MediaWiki extension is the easiest implementation form for the review system: it will make integration with Wikipedia user accounts easier, and review data will be stored locally. The Article feedback tool provides an existing framework that could be extended to support a more detailed quality review process.

Authentication

 * How does a credentialed reviewer authenticate?
 * How can the authentication process be scaled / automated? (use case: mass outreach from organizations to their members)

E-mail is the safest assumption we can make on the partner organization's infrastructure.

Organization members would get an invitation by e-mail to create an account on Wikipedia; the token/link would lead them to a modified version of the sign-up page, where they would confirm their organization, real name, e-mail address, and possibly volunteer to list their credentials and/or field of expertise.

If the partner organization agrees to provide us with a structured document containing this information (e.g. a CSV file), a script could be run to generate these e-mail invitations.

If the partner organization prefers not to share this information with us, we would provide them with a modified script that would only identify the organization; their members would then enter the rest of the information themselves.

The reviewer should be able to attach their existing Wikipedia account if they have one, instead of creating a new one.

Review submission

 * Who decides who reviews what?
 * How is the review submitted?

A voluntary model, where people can review any page they want, is the simplest implementation, and the most likely to fit within the existing article feedback infrastructure. Restricting the scope of reviewable articles doesn't appear to be necessary: existing expert review systems show that reviewers usually stick to their field of expertise when their name is publicly associated with the review.

Because articles can grow fairly long, it would be better to allow the reviewer to scroll through the article while they're reviewing it, while keeping the review fields always visible (some suggested a setup similar to common the fixed-position "feedback tab").

Review content

 * What is the content of the actual review?

Analysis of the Article feedback experiment shows that Wikipedians have a consistent grasp of what criteria like "neutrality" and "well-sourced" mean, and rate them fairly consistently. The general public, however, doesn't have the same model.

As a consequence, it's probably better to ask specific questions that everyone can understand.

Possible questions include:
 * "Would you recommend this article to a friend, a student, or for inclusion in a book?"


 * Binary flag
 * Likert scale (and similar)
 * Problem tagging (à la EOL)
 * Free-text comment
 * Guided questions
 * Article feedback/UX Research
 * reviewer's confidence?


 * Reviewers are invited to edit the page they're reviewing and fix the errors they notice.
 * Reviewers are invited to disclose conflicts of interest about the article itself (e.g. the reviewer has significantly edited the article) or the topic (e.g. the reviewer is a known critic of some of the points made); if their reviewer account is the same as, or linked to, their Wikipedia user account, they are prompted to disclose an article COI if they've edited it.

Review publication

 * Where is the review published?

Reviews are stored in a database and can be accessed via the article page, possibly through an additional tab. A visual indicator could give an overview of the content of the reviews and thresholds of serious issues (or lack thereof).

Readers should have the ability to sort or filter the list of reviews for a given article, for example by date (to show the latest review first) or by reviewer (to show "experts" first).


 * moderation
 * Public list of one's reviews
 * API to access the entirety of the reviews and their specifics

API

 * standards / policies for integration of data from external review systems with ours