Article feedback/Version 5

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:
 * Useful links to prototypes and reports
 * Feature Requirements
 * First Report on Article Feedback V5 Results
 * Sample Articles for Testing
 * Metrics Dashboard
 * Data and Metrics Plan
 * Interactive prototype
 * Technical Design
 * Wikipedia article on AFT5
 * Wikipedia talk page

Goals
In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. To that end, the foundation hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

The overall goal of this project is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods. We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:
 * engage readers to participate more on Wikipedia
 * give editors new tools to improve article quality
 * encourage readers to become editors over time
 * invite a collaboration between editors and readers
 * experiment with outsourcing web development

The first implementations of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. A new Version 5 of the Article Feedback Tool (AFT V5) is now being tested, to take into account both community feedback and data analysis for the current version (AFTv4). The main goal is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods.

For Version 5, the Foundation is replacing the current ratings system with "comment" boxes, to invite constructive suggestions for improvement, rather than judgmental ratings. A reader would be able to provide useful feedback (such as "you're missing a section on that person's childhood") that editors could use to improve the article (as opposed to simply rating the article "3 stars" for completeness). This would allow readers to better engage with editors, who could then actively improve articles if they find the feedback helpful.

The first option that was tested for Version 5 (see image) was a simple feedback form that asks the readers: "Did you find what you were looking for?", with 'Yes' or 'No' buttons. Readers are then invited to write a comment and suggest improvements to this article. This approach may provide implicit measurements of quality that could be as effective as explicit ratings in the current form. For example, with the "Did you find what you're looking for" question, the percentage of "yes"'s could be an indicator of article quality.

Another feedback form that was tested for V5 enabled users to post different types of feedback: make a suggestion, ask a question, report a problem or give praise to the authors. A third option was also tested, which allowed users to rate articles and add a comment. Test results showed a slight overall preference for Option 1 (even though the other two options were also found useful). For that reason, the first option was selected for the next phase of development and new variations of that feedback form will undergo further testing. To learn more, read the first report about the first phase of this experiment.

These new article feedback forms are being tested on a small group of randomly-selected articles (0.6 percent of the English Encyclopedia). The Wikimedia Foundation is collecting detailed metrics for each of these feedback formats and is working with the community to evaluate their effectiveness. As usual, the relevant data will be made public so that anyone can dig into the numbers.

A new feedback page will list all the user posts for each article. All users will have access to this feedback page, and they will be able to filter or sort that list in a variety of ways. Experienced editors and administrators will have the option to feature posts more prominently, or hide offensive posts. Tools enabling the community to mark posts as helpful will also be available, along with the ability to flag abuse. The actual nature of the comments we receive from readers (e.g., signal-to-noise ratio, what users actually comment on, etc.) will influence the design of the feedback page, as well as recommendations from the editor community.

For a preview of what some of these features might look like, check the for Version 5. (Some of these slides are for discussion purposes only, and the final versions are likely to be different from these early visualizations.)

Features


Key features for AFT V5 include: (see also full feature requirements page)
 * new feedback forms
 * calls to action
 * feedback page
 * monitoring tools

In the first phase of this project (Dec.-Feb. 2011), we developed and started testing three different feedback forms to extend the current rating tool: We tested these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see [our report on first test results).
 * Option 1: Did you find what you were looking for?
 * Option 2: Make a suggestion
 * Option 3: Review this page

In the next phases of this project, we plan to develop and test these additional options: In the spring of 2012, we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.
 * Feedback links
 * New calls to action
 * A stand-alone call to edit
 * Feedback page

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this.

Get involved
We are developing this new tool in collaboration with Wikipedia editors, with whom we meet regularly over IRC and other channels. We are looking for more volunteers to help improve this article feedback tool.

Talk page
You're invited to give feedback about the Foundation's current plans for Version 5 and let us know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats
To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule below. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- Jan. 6, Jan. 13, Jan. 20 , Jan. 27 and Feb. 16, 2012.

Feedback Evaluation
The Foundation is looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools in 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained to review feedback posts for this evaluation.

Schedule
The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project. All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Weekly Update
Last week: (May 14)
 * Developed, tested and deployed new design for feedback page and permalink page
 * Developed, tested and deployed central feedback page (non-public URL)
 * Tested and deployed more bug fixes and tweaks to new feedback page features (e.g. Relevance)
 * Deployed and tested new version of Abuse filter (new code for group filters)
 * Removed feedback links (Option A or E), after Metrics Stage 3 data collection was complete
 * Started development on final feedback form (based on Option 1, progressive reveal)
 * Prepared Metrics Stage 4 click tracking & impressions for feedback page tests
 * Discussed options for remaining features (throttling, CTA4, checkuser, help pages, my contribs, etc.)
 * Prepared test plan with Chris McMahon for volunteer testing with OpenHatch on June 9th

This week: (May 21)
 * Test and deploy CTA 5 (feedback page link) and talk page link
 * Test and deploy Metrics Stage 4 clicktracking for feedback page tests
 * Test and deploy survey button ("What do you think?") and link to Survey Monkey
 * Deploy more bug fixes and tweaks for new feedback page features
 * Continue development of final feedback form (based on Option 1, progressive reveal)
 * Define and develop solution for highlighting your last post on feedback page
 * Invite editors to create more abuse filters, resolve questions about warning message, vandalism bug
 * Write and edit help pages for new feedback page features (editors, monitors, oversighters)
 * Prepare for follow-up IRC chat and wider socialization the week of May 28?
 * Discuss and prepare feedback page usability tests with editors (user testing.com)
 * Finalize requirements and estimates for remaining features (Throttling, Checkuser, Help pages, MyContribs, etc.)
 * Finalize OmniTI contract extension, plan next steps through hand-off
 * Start planning mobile deployment

All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Metrics
AFT v.5
 * For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
 * Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

AFT v.4
 * We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
 * Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
 * Global daily ratings and conversions
 * Volume of daily ratings per article
 * Most frequently rated articles
 * Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team
Here are the Wikimedia team members who are working on this project at this time:
 * VP Engineering / Product	Erik Moeller
 * Senior Product Manager	Howie Fung
 * Product Manager, Editor Engagement	Fabrice Florin (project lead and primary author of this page}
 * Features Engineering Director	Alolita Sharma
 * Senior Research Analyst	Dario Taraborelli
 * UX Designer	Brandon Harris
 * Code Review / Testing	Roan Kattouw
 * Community Outreach	Oliver Keyes
 * Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers Here are the team members from OmniTI, our development partner:
 * VP Business Development  Leon Fayer
 * Project Manager  Yoni Shostak
 * Developer  Reha Sterbin
 * Developer  Greg Chiasson
 * UI Developer/Designer  Sean Heavey

Community members Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:
 * Bensin
 * Dougweller
 * Looie496
 * RJHall
 * Utar

We’d like to give special thanks and recognition to these community members who have already done so much to develop this new tool: we are grateful for your insights and commitment to this cause!