Quality review
From MediaWiki.org
 |
This page is currently a draft. Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page. |
The open nature of wikis makes quality management a serious challenge. This page gives a general overview of quality review systems and their characteristics.
Contents |
[edit] Review processes by types
A review process can be:
- internal: the process is mainly handled locally by the wiki's community, and the data is stored in the wiki's pages or its database;
- external: the process is mainly handled by an external group, and the data is stored outside the wiki. Integration with the wiki is optional.
- mixed: e.g., an external group posts reviews on the wiki.
Some authority-based review processes distinguish between "experts" and others. In this case, reviewers can be classified whether they are:
- "non-experts"
- people self-identifying as experts (no credentials verification)
- people identifying as experts whose credentials have been verified.
[edit] Example review processes
- A more detailed description is available for some projects.
| Review process | Wiki | Type | Reviewers | Review location | Review tool | Visibility on page | Notes |
|---|---|---|---|---|---|---|---|
| WikiProject assessments | Wikipedia | internal | local community (thematic "WikiProjects") | wiki talk pages | optional (JS) | optional (JS) | |
| WikiProject Medicine / Google | Wikipedia | mixed | professional medical editors | wiki talk pages | no | no | Example |
| Public Policy Initiative | Wikipedia | mixed | local community and subject matter experts | Google document | no | no | |
| Article feedback pilots | Wikipedia | mixed | anyone (readers & authors) | wiki database | yes (MW ext.) | yes | |
| Encyclopedia of Life | Wikipedia | external | Individual EOL curators | external database & Wikimedia toolserver | ? | optional (JS) | |
| Rfam / Pfam | Wikipedia | external | academics from the Wellcome Trust Sanger Institute[1] | external database? & Wikimedia Toolserver | ? | optional (JS) | |
| APS Wikipedia Initiative | Wikipedia | external | technically, anyone, but filtered a posteriori[2] | external database | yes (PHP) | no | Harnessing the Power of Wikipedia for Scientific Psychology: A Call to Action |
- ↑ People with academic degrees or publications in the relevant science (RNA/protein biology)
- ↑ Reviews are filtered a posteriori based on some basic level information provided by reviewers about their education level and whether they are a member of APS or not.
[edit] Review systems
| Review process | Type[1] | Binary flag[2] | Metrics | Metrics values | Free-form comments allowed | Notes |
|---|---|---|---|---|---|---|
| WikiProject assessments | exclusive | no | quality, importance | quality: 7 (stub, start, C, B, GA, A, FA); importance: 4 (low, mid, high, top) | yes | |
| WikiProject Medicine / Google | cumulative system, but exclusive in practice | no | none (qualitative) | n/a | yes | checked against good article criteria |
| Public Policy Initiative | no | comprehensiveness, sourcing, neutrality, readability, formatting, illustrations | resp. 1-10, 0-6, 0-3, 0-3, 0-2, 0-2 | |||
| Article feedback pilots | cumulative | no | well-sourced, neutral, complete, readable | 1-5 (stars) | no | possibility to skip metrics (value = 0) |
| Encyclopedia of Life | exclusive | yes (trusted/untrusted) | misidentified, incorrect/misleading information, poor writing/image/sound quality, redundant/duplicate, other | 2 (boolean) | yes | user flags issues rather than assessing metrics |
| Rfam / Pfam | ||||||
| APS Wikipedia Initiative | cumulative | no | trustworthy, unbiased, complete, well-written, accurate | 5 (strongly disagree / disagree / neutral / agree / strongly agree) | yes | |
| Article feedback extended review | cumulative | TBD | TBD | TBD | yes | planning phase |
- ↑ Exclusive: only one master review/rating possible; cumulative: multiple reviews/ratings possible.
- ↑ For example to select articles for publication in other media like books.
[edit] User research
In the context of quality review, there are two kinds of users: reviewers and review readers.
From a reviewer point of view
- What system, how it works, what the goals are, what they expect
- Do they want to communicate with review readers?
From a review reader point of view
- What are their goals by reading a review? (i.e. what is useful for them)
- What do they expect from a review?
- Do they want to communicate with reviewers?
People who have already created review processes might be able to explain:
- what were their goals when they created these tools & processes
- whether some design decisions were intentional (e.g. type/scale of rating)
[edit] Domain research
Sources:
- Article feedback/UX Research
- Quality assessment tools for Wikipedia readers
- Interviews
- m:Expert review & m:Talk:Expert review
- Survey: Expert barriers to Wikipedia (includes relevant questions such as: (1) whether respondents volunteer to be interviewed; (2) whether they would be willing to review wiki articles)
