Article feedback/Version 5

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:
 * Useful links to prototypes and reports
 * Feature Requirements
 * First Report on Article Feedback V5 Results
 * Sample Articles for Testing
 * Metrics Dashboard
 * Data and Metrics Plan
 * Interactive prototype
 * Technical Design
 * Wikipedia article on AFT5
 * Wikipedia talk page

Goals
In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. To that end, the foundation hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

The overall goal of this project is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods. We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:
 * engage readers to participate more on Wikipedia
 * give editors new tools to improve article quality
 * encourage readers to become editors over time
 * invite a collaboration between editors and readers
 * experiment with outsourcing web development

The first implementations of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. A new Version 5 of the Article Feedback Tool (AFT V5) is now being tested, to take into account both community feedback and data analysis for the current version (AFTv4). The main goal is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods.

For Version 5, the Foundation is replacing the current ratings system with "comment" boxes, to invite constructive suggestions for improvement, rather than judgmental ratings. A reader would be able to provide useful feedback (such as "you're missing a section on that person's childhood") that editors could use to improve the article (as opposed to simply rating the article "3 stars" for completeness). This would allow readers to better engage with editors, who could then actively improve articles if they find the feedback helpful.

The first option that was tested for Version 5 (see image) was a simple feedback form that asks the readers: "Did you find what you were looking for?", with 'Yes' or 'No' buttons. Readers are then invited to write a comment and suggest improvements to this article. This approach may provide implicit measurements of quality that could be as effective as explicit ratings in the current form. For example, with the "Did you find what you're looking for" question, the percentage of "yes"'s could be an indicator of article quality.

Another feedback form that was tested for V5 enabled users to post different types of feedback: make a suggestion, ask a question, report a problem or give praise to the authors. A third option was also tested, which allowed users to rate articles and add a comment. Test results showed a slight overall preference for Option 1 (even though the other two options were also found useful). For that reason, the first option was selected for the next phase of development and new variations of that feedback form will undergo further testing. To learn more, read the first report about the first phase of this experiment.

These new article feedback forms are being tested on a small group of randomly-selected articles (0.6 percent of the English Encyclopedia). The Wikimedia Foundation is collecting detailed metrics for each of these feedback formats and is working with the community to evaluate their effectiveness. As usual, the relevant data will be made public so that anyone can dig into the numbers.

A new feedback page will list all the user posts for each article. All users will have access to this feedback page, and they will be able to filter or sort that list in a variety of ways. Experienced editors and administrators will have the option to feature posts more prominently, or hide offensive posts. Tools enabling the community to mark posts as helpful will also be available, along with the ability to flag abuse. The actual nature of the comments we receive from readers (e.g., signal-to-noise ratio, what users actually comment on, etc.) will influence the design of the feedback page, as well as recommendations from the editor community.

For a preview of what some of these features might look like, check the for Version 5. (Some of these slides are for discussion purposes only, and the final versions are likely to be different from these early visualizations.)

Features


Key features for AFT V5 include: (see also full feature requirements page)
 * new feedback forms
 * calls to action
 * feedback page
 * monitoring tools

In the first phase of this project (Dec.-Feb. 2011), we developed and started testing three different feedback forms to extend the current rating tool: We tested these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see [our report on first test results).
 * Option 1: Did you find what you were looking for?
 * Option 2: Make a suggestion
 * Option 3: Review this page

In the next phases of this project, we plan to develop and test these additional options: In the spring of 2012, we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.
 * Feedback links
 * New calls to action
 * A stand-alone call to edit
 * Feedback page

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this.

Get involved
We are developing this new tool in collaboration with Wikipedia editors, with whom we meet regularly over IRC and other channels. We are looking for more volunteers to help improve this article feedback tool.

Talk page
You're invited to give feedback about the Foundation's current plans for Version 5 and let us know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats
To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule below. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- Jan. 6, Jan. 13, Jan. 20 , Jan. 27 and Feb. 16, 2012.

Feedback Evaluation
The Foundation is looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools in 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained to review feedback posts for this evaluation.

Schedule
The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project. All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Weekly Update
Last week: (May 7)
 * Developed and tested new design for feedback page and permalink page
 * Develop and tested central feedback page (prototype)
 * Deployed bug fixes and tweaks to new feedback page features (Relevance)
 * New version of Abuse filter posted for review (new code for group filters)
 * Developed CTA 5 (feedback page link) and talk page feedback link
 * Prepared Metrics Stage 4 click tracking & impressions for feedback page tests
 * Prepare Metrics Stage 4 survey second draft, with legal approval
 * Finalized final feedback form design (expandable version of Option 1)
 * Posted Bugzilla tickets for upcoming features (throttling, CTA4, checkuser, help pages, my contribs, etc.)
 * Worked with Chris McMahon to set up volunteer testing with OpenHatch on June 9th
 * Negotiated and signed OmniTI amendment extending contract through end of June

This week: (May 14)
 * Develop, test and deploy new design for feedback page and permalink page
 * Develop, test and deploy central feedback page (non-public URL)
 * Test and deploy more bug fixes and tweaks to new feedback page features (e.g. Relevance)
 * Deploy and test new version of Abuse filter (new code for group filters)
 * Test and deploy CTA 5 (feedback page link) and talk page feedback link
 * Develop and deploy Metrics Stage 4 click tracking & impressions for feedback page tests
 * Deploy Metrics Stage 4 survey and 'What do you think?' button on feedback page
 * Remove feedback links (Option A or E), once Metrics Stage 3 data collection is done
 * Finalize requirements and estimates for remaining features (throttling, CTA4, checkuser, help pages, my contribs, etc.)
 * Discuss and prepare feedback page usability tests
 * Write and edit help pages for new feedback page features
 * Prepare test plan with Chris McMahon for volunteer testing with OpenHatch on June 9th

All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Metrics
AFT v.5
 * For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
 * Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

AFT v.4
 * We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
 * Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
 * Global daily ratings and conversions
 * Volume of daily ratings per article
 * Most frequently rated articles
 * Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team
Here are the Wikimedia team members who are working on this project at this time:
 * VP Engineering / Product	Erik Moeller
 * Senior Product Manager	Howie Fung
 * Product Manager, Editor Engagement	Fabrice Florin (project lead and primary author of this page}
 * Features Engineering Director	Alolita Sharma
 * Senior Research Analyst	Dario Taraborelli
 * UX Designer	Brandon Harris
 * Code Review / Testing	Roan Kattouw
 * Community Outreach	Oliver Keyes
 * Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers Here are the team members from OmniTI, our development partner:
 * VP Business Development  Leon Fayer
 * Project Manager  Yoni Shostak
 * Developer  Reha Sterbin
 * Developer  Greg Chiasson
 * UI Developer/Designer  Sean Heavey

Community members Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:
 * Bensin
 * Dougweller
 * Looie496
 * RJHall
 * Utar

We’d like to give special thanks and recognition to these community members who have already done so much to develop this new tool: we are grateful for your insights and commitment to this cause!