Article feedback/Version 5/Report

Overview
A new experimental version 5 of the Article Feedback Tool (AFT5) was developed by the Wikimedia Foundation from November 2011 to June 2013. The main purpose of this experiment was to increase participation on Wikipedia by inviting readers to leave comments on article pages. To that end, we tested an updated version of this tool in three different pilots on the English, French and German Wikipedias. (see pilot results below)

Article Feedback seemed effective for engaging readers: 70% of survey respondents liked the tool; after leaving feedback -- and. Over a million comments were posted during this experiment: on average, 12% of posts were marked as useful, 46% required no action, and 17% were found inappropriate by Wikipedia editors.

However, a majority of editors did not find reader comments useful enough to warrant the extra work of moderating this feedback. The French pilot just ended last month, providing further confirmation of this issue. In the final RfC we ran on the French site, about 45% of respondents wanted AFT5 removed everywhere, while 38% wanted to keep it on an opt-in basis, and 10% on help pages only; nearly everyone agreed it should not be on by default on all 40,000 pilot pages, let alone on the entire French Wikipedia. Their concerns are consistent with what we heard from editors on the English and German pilots.

On the English Wikipedia, Article Feedback is now available on an opt-in basis, and has been enabled on about 3,677 articles by editors who want reader feedback on articles they watch. On average, editors enable this tool on about 100 new articles per week -- but some editors who oppose the tool have disabled it on thousands of articles in recent months, which creates an awkward tension in our community. (Previously, the tool was enabled on over 400,000 articles, and 672k comments were collected during this pilot; but about 63% of editors in a February 2013 RfC voted against wide deployment, leading to the removal of the tool on these articles.)

In a retrospective we conducted in early February with team members involved in the development of Article feedback (see details below), the consensus was that the time has come for us to retire this tool. Most participants agreed that Flow is better positioned to give our readers a voice -- and that we should clear the way to make it a success.

Recommendation
Based on these pilot results and our team retrospective below, we recommend that Article Feedback be removed from all pilot sites, as it is clearly not welcome by a majority of editors, despite its benefits to readers.

Flow, our new discussion tool, is expected to offer a more practical alternative for reader feedback later this year. In the long-term, it is likely to provide a more effective solution than the experimental Article Feedback tool -- which was hampered by a lack of development resources and was never well received by our editor community, due to a series of ineffective trials over the years.

Though some editors expressed interest in keeping the tool on an opt-in or limited basis, Article Feedback would need significant improvements to better serve its users, and the foundation doesn't have the resources to develop it further at this time. Besides being unpopular with many in our editor community, it is also slowing down site performance -- and may require more technical maintenance that we can adequately provide.

As a result, we recommend this tool be retired from our two pilot sites, so we can focus our limited resources on Flow and other projects instead. This recommendation aligns with what we've heard from community and team members, after an extensive consultation period. Highlights of our team retrospective are also included below, for more perspectives on this experiment.

Update: The Article Feedback Tool was removed on March 3, 2014 from both the English and French Wikipedia sites, as reported here. In coming days, we will archive the feedback data in a public hub, so it can be accessed without the tool. We will post on this thread as soon as that data archive is available. Thanks again to everyone who contributed to this experiment -- we're grateful for your support of this project. To be continued. Fabrice Florin (WMF) (talk) 02:27, 4 March 2014 (UTC)

Next steps
We propose these follow-up actions for this project.

Announcement
We plan to announce this recommendation to the English and French communities on February 12, 2014.

Tool Removal
We propose to remove AFT5 entirely from both enwiki and frwiki sites at the end of February, on a date to be determined based on community feedback and team availability.

Transition Period
We propose to give editors two weeks to transfer any feedback they find useful to their article talk pages (with the ‘Discuss on talk page’ tool).

Archives
We plan to preserve the reader feedback from the pilot sites in compact text files, so that researchers could access it for research purposes.

Prototype site
For reference purposes, we also plan to keep one instance of the tool running on a Labs prototype site, subject to team availability to set this up.

Other projects
If other projects wish to enable AFT5 on their sites, their communities are welcome to do this on their own, but the foundation cannot provide active support for this project, so this would have to be a purely volunteer initiative.

Discussions
Lastly, we recommend further discussions between the community and the foundation on how to give a voice to readers on our sites, with a focus on these topics:

* How can we make it easier for readers to comment on articles they read?

Casual users find talk pages and editing tools hard to use -- and their comments are not welcomed by many editors. The first issue is mostly technical and could be solved over time with new tools like Flow. But the second issue is social and may require a change in behavior to better integrate newcomers into the current Wikipedia culture.

* How can readers participate in decisions that impact them?

Typically, readers are not well represented in editor polls, RfCs and other on-wiki discussions about new features that concern them. In the case of AFT, only a few hundred editors voted to remove a tool that could have benefited hundreds of millions of readers -- and this decision was reached without any real representation for this important user group. This doesn't seem right for a site that claims to be for everyone. How could we solve this?

We look forward to discussing these important questions in coming months, so that we can better serve the needs of all our users in the free culture movement. In the meantime, we wish to thank all of the community and team members who contributed to this experiment.

Retrospective
In early February, we invited team members who participated in this project to share recommendations for next steps, as well as lessons they learned during the planning, development and pilot testing phases. The goal of this retrospective was to finalize our recommendation, identify specific strengths and weaknesses of this experiment, as well as offer suggestions that could help future projects.

Here are highlights of lessons learned by team members who contributed:

What did we learn about this product?

 * Commenting systems are hard to do
 * Nobody likes a pile of homework
 * The impetus to comment for readers is different than the impetus to read for editors

What did we do well?

 * Community engagement helped improve the tool
 * Many technical improvements, which are now being used elsewhere


 * Web analytics and engagement study tools

What did we not do so well?

 * Insufficient development resources slowed us down
 * With products that require community buy-in, simply talking is not enough
 * Should have experimented with the core concepts earlier


 * Some scope creep

How can we do better next time?

 * Clearly frame the problem we aim to solve


 * Validate our assumptions about community behavior as early as possible


 * Involve as many communities as possible (not just English Wikipedia)


 * Resist our convictions / Do not defend

How can we give readers a voice on our sites?

 * Provide a good survey tool
 * Limited feedback per editor on demand
 * Give readers a voice, but keep the signal-to-noise ratio high
 * Give editors a voice when designing feedback tools
 * Give visibility to the talk page

See internal retrospective document for more details.

Pros and Cons
Here are some of the pros and cons we considered when evaluating next steps for AFT5:

Pros

 * effective as a reader engagement tool
 * gives a voice to visitors who are not ready to edit yet
 * increases participation, converts some readers into editors
 * posting feedback increases registration (with a 2.7% registration rate)
 * increases customer satisfaction (70% of readers like the tool)
 * tracks article quality (how many 'found what they were looking for')

Cons

 * most of the feedback is not useful to editors (~12% of posts seem useful)
 * moderating comments creates more work for editors
 * no development resources to improve the tool
 * some maintenance required for this complex software tool
 * some impact on site performance
 * distracts product teams from priority goals

Pilot Results
The final version of AFT5 was tested on 3 pilot sites from March to December 2013. Here are key findings for each pilot.

English Wikipedia Pilot

 * had been enabled on 400,000 articles through March 2013
 * now enabled by editors on 3,677 articles, on an opt-in basis
 * increasing at the rate of ~100 new articles/week
 * feedback was disabled on thousands of articles by editors opposing this tool
 * 672k comments posted as of Feb. 12 (plus 259k posts without comments, 932k total)
 * 65k of these comments were featured or resolved (~10%)
 * 560k posts were unreviewed, 46k required no action, 8k were found inappropriate
 * about 175 editors participated in February 2013 RfC
 * about 63% voted against wide deployment then
 * AFT5 discussion page includes ongoing concerns about this tool

French Wikipedia Pilot

 * was enabled on 40k articles until Feb. 9, 2014
 * 165 editors participated in first pilot poll
 * 62% voted in favor of feedback pilot
 * half voted for wide release on all articles
 * 6-month pilot started in June 2013, ended in January 2014
 * 223k comments posted as of Feb. 12 (plus 46k posts without comments, 270k total)
 * 25k comments were featured/resolved (~11%)
 * 56k posts were unreviewed, 124k required no action, 18k were found inappropriate
 * 103 editors participated in December 2013 survey
 * 60% wanted to remove AFT5, 36% wanted to keep opt-in version
 * 91 editors participated in January 2014 poll
 * 45% wanted AFT5 removed everywhere, 38% wanted it kept on opt-in basis, 10% on help pages only
 * everyone agreed it should not be on by default on all 40k pilot pages, so it was removed on most articles
 * a dozen projects wanted AFT5 kept on their pages, where the tool is still enabled

German Wikipedia Pilot

 * was enabled on 13,000 articles for 6 months
 * 226 editors participated in poll in May 2013
 * 65% voted against article feedback
 * feedback tool was removed

Moderation Stats
Average stats across all 3 pilot sites:
 * Useful 	12%
 * Resolved	4%
 * No action needed	46%
 * Inappropriate	17%

Impact on Readers

 * 70% of survey respondents liked the feedback tool (mostly readers)
 * 2.7% of invited readers registered after leaving feedback
 * 3.0% of invited readers completed an edit

Related Pilot Links

 * Slides (see pilot slides)
 * Wikimania 2013 Presentation (YouTube video)
 * Feedback Stats (spreadsheet comparison)
 * Feedback Moderation Dashboards (enwiki, frwiki, dewiki)
 * Feedback Volume Dashboards (enwiki, frwiki, dewiki)
 * Sample feedback (Golden-crowned Sparrow, Government Shutdown articles)

Wikipedia Article Feedback corpus
The full dataset can be downloaded.

"The Article Feedback experiment invited readers to participate on Wikipedia by leaving comments on articles, to help editors improve them. This data set includes over 1.5 million messages posted to the English, French and German Wikipedia during the pilot."
 * http://datahub.io/dataset/wikipedia-article-feedback-corpus
 * (or this duplicate) http://figshare.com/articles/Wikipedia_Article_Feedback_corpus/1277784

It was previously mentioned in https://blog.wikimedia.org/2015/03/13/open-data-sets/