Edit Review Improvements

Edit Review Improvements is a project of the Collaboration Team, which is researching ways to reduce the negative effects current edit-review process can have on new editors to the wikis. Most edit-review and patrolling tools were designed to safeguard content quality and fend off bad actors—both vitally important missions. A large body of research, however, suggests that current edit-review process, and use of automated or semi-automated tools in particular, can have the unintended consequence of discouraging and even driving away good-faith new editors.

To solve this problem, Collaboration Team is investigating ways to separate good-faith new users from current edit-review workflows and, ultimately, to provide a supportive review process that helps new users become productive contributors.

Problem

 * Research shows that for new wiki editors in particular, “being reverted predicts both a decrease in activity and a reduction in the probability of survival” as editors.
 * At the same time, the increasing use of automated and semi-automated edit-review tools has brought about an increase in rejection of good-faith newcomers.  The use of these tools “significantly increases the negative effect of rejection on desirable newcomer retention.”
 * The above notwithstanding, edit-review tools are essential for vandalism fighters and others working to maintain wiki integrity and quality. How can we help and retain new users while maintaining the productivity of vandalism fighters and other edit reviewers?

Goals
Ultimately this project aims to have an effect on editor retention, an objective that aligns well with the Wikimedia Foundation 2016-17 Annual Plan, developed in close consultation with the community. The approach also tracks with the goals the Annual Plan lays out for the Product Team, which promise, among other things, to “Invest in new types of content…curation and collaboration tools.”
 * Ensure good-faith new editors have more constructive, less discouraging experiences of edit and article review.
 * By providing richer data about recent changes, enable patrollers and edit-reviewers of all types to work more efficiently and to pursue diverse interests (e.g., fighting vandalism, supporting new users)  in a more effective and targeted way.

Solutions
This project is in a research phase and no concrete product plans have been formalized as of this writing (in June, 2016). To begin to address the problems of struggling but good-faith newcomers, however, a good first step will be to ensure that we can find them. The Collaboration Team’s quarterly goal for the first quarter of 2016-2017 (July-September 2016) is to “create a process that enables edit-reviewers to identify the edits of good-faith new users, so that these edits can be reviewed separately.”

To achieve this, we propose to analyze recent changes using data from a variety of sources, including and most notably the machine-learning program ORES (Objective Revision Evaluation Service). ORES’s good faith model, trained on human judgement, is highly accurate at finding good-faith edits (it can find 95% of good-faith edits with 98% accuracy). ORES can also predict edits that will be reverted and those that are damaging to the wikis.

While research shows that new editors are particularly vulnerable to rejection, there’s also evidence that edit-review and even rejection can be a powerful learning experience for newcomers. For reviewers interested in supporting new users, a stream of edits that are a) likely to be reverted but which were b) made in good faith, will represent, we hope, a string of teachable moments.

The edit analysis described above may be presented to users in a number of ways, including:
 * On a dedicated page similar to w:en:Special:NewPagesFeed.
 * On Recent Changes pages via a new filter or filters.
 * In machine-readable feeds that can be ingested by edit-review programs.