Article feedback/Version 5

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:
 * Useful links to prototypes and reports
 * Feature Requirements
 * First Report on Article Feedback V5 Results
 * Sample Articles for Testing
 * Metrics Dashboard
 * Data and Metrics Plan
 * Interactive prototype
 * Technical Design
 * Wikipedia article on AFT5
 * Wikipedia talk page

Goals
In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. To that end, the foundation hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

The overall goal of this project is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods. We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:
 * engage readers to participate more on Wikipedia
 * give editors new tools to improve article quality
 * encourage readers to become editors over time
 * invite a collaboration between editors and readers
 * experiment with outsourcing web development

The first implementations of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. A new Version 5 of the Article Feedback Tool (AFT V5) is now being tested, to take into account both community feedback and data analysis for the current version (AFTv4). The main goal is to find ways for readers to contribute productively to building the encyclopedia. Version 5 will also continue to test different methods for measuring quality, through both implicit and explicit methods.

For Version 5, the Foundation is replacing the current ratings system with "comment" boxes, to invite constructive suggestions for improvement, rather than judgmental ratings. A reader would be able to provide useful feedback (such as "you're missing a section on that person's childhood") that editors could use to improve the article (as opposed to simply rating the article "3 stars" for completeness). This would allow readers to better engage with editors, who could then actively improve articles if they find the feedback helpful.

The first option that was tested for Version 5 (see image) was a simple feedback form that asks the readers: "Did you find what you were looking for?", with 'Yes' or 'No' buttons. Readers are then invited to write a comment and suggest improvements to this article. This approach may provide implicit measurements of quality that could be as effective as explicit ratings in the current form. For example, with the "Did you find what you're looking for" question, the percentage of "yes"'s could be an indicator of article quality.

Another feedback form that was tested for V5 enabled users to post different types of feedback: make a suggestion, ask a question, report a problem or give praise to the authors. A third option was also tested, which allowed users to rate articles and add a comment. Test results showed a slight overall preference for Option 1 (even though the other two options were also found useful). For that reason, the first option was selected for the next phase of development and new variations of that feedback form will undergo further testing. To learn more, read the first report about the first phase of this experiment.

These new article feedback forms are being tested on a small group of randomly-selected articles (0.6 percent of the English Encyclopedia). The Wikimedia Foundation is collecting detailed metrics for each of these feedback formats and is working with the community to evaluate their effectiveness. As usual, the relevant data will be made public so that anyone can dig into the numbers.

A new feedback page will list all the user posts for each article. All users will have access to this feedback page, and they will be able to filter or sort that list in a variety of ways. Experienced editors and administrators will have the option to feature posts more prominently, or hide offensive posts. Tools enabling the community to mark posts as helpful will also be available, along with the ability to flag abuse. The actual nature of the comments we receive from readers (e.g., signal-to-noise ratio, what users actually comment on, etc.) will influence the design of the feedback page, as well as recommendations from the editor community.

For a preview of what some of these features might look like, check the for Version 5. (Some of these slides are for discussion purposes only, and the final versions are likely to be different from these early visualizations.)

Features


Key features for AFT V5 include: (see also full feature requirements page)
 * new feedback forms
 * calls to action
 * feedback page
 * monitoring tools

In the first phase of this project (Dec.-Feb. 2011), we developed and started testing three different feedback forms to extend the current rating tool: We tested these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see [our report on first test results).
 * Option 1: Did you find what you were looking for?
 * Option 2: Make a suggestion
 * Option 3: Review this page

In the next phases of this project, we plan to develop and test these additional options: In the spring of 2012, we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.
 * Feedback links
 * New calls to action
 * A stand-alone call to edit
 * Feedback page

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this.

Get involved
We are developing this new tool in collaboration with Wikipedia editors, with whom we meet regularly over IRC and other channels. We are looking for more volunteers to help improve this article feedback tool.

Talk page
You're invited to give feedback about the Foundation's current plans for Version 5 and let us know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats
To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule below. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- Jan. 6, Jan. 13, Jan. 20 , Jan. 27 and Feb. 16, 2012.

Feedback Evaluation
The Foundation is looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools in 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained to review feedback posts for this evaluation.

Schedule
The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project. All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Weekly Update
Last week: (June 11)


 * Developed and deployed a range of features and bug fixes based on community/testing feedback
 * Started development on remaining features (CTA4, My Last Post, My Contributions, etc.)
 * Discussed plan for increasing article sample size for wider deployment
 * Wrapped up Metrics Stage 5 clicktracking (Option 1/CTA3 vs. Option 6/CTA5)
 * Removed CTA3 linking to new survey 6 for feedback form (Option 6 / CTA 3)
 * Developed and tested new abuse filters on en-wiki (to disallow or flag obscenities in feedback)
 * Tweaked final feedback form design (Option 6, progressive reveal)
 * Tweaked talk page link to be closer to title
 * Discussed adding Moodbar reply feature to AFT with Benny + Matthias for later this summer
 * Edited and published help FAQ pages for all user groups (inc. monitors, oversighters)
 * Discussed wider socialization plan first draft (Oliver)
 * Continued OmniTI training and documentation for hand-off in late June
 * Made wider release decisions in project review meeting

This week: (June 18)


 * Test and deploy My Last Post (show it on top of feedback page, with New label)
 * Test and deploy updated feedback form design (Option 6, progressive reveal)
 * Test and deploy CTA 4 (Create an account or login, to increase registered members)
 * Develop, test (and deploy?) code to increase article sample size for wider deployment
 * Develop, test (and deploy?) feedback on My Contributions (includes changes to core)
 * Develop and test IE8 fixes and other configuration issues
 * Develop and test more features and bug fixes based on community/testing feedback
 * Develop, test (and deploy?) default warning message for Abuse filter (Andrew) + new abuse filters
 * Tweak talk page link to be closer to title (hold release until July 3)
 * Prioritize Bugzilla bug list with development team members
 * Get Jeremy Postlethwaite started on unit testing and OmniTI knowledge transfer
 * Set up scalability meeting for AFT5 with key members of engineering team
 * Update help FAQ pages links and graphics for all user groups (e.g. monitors, oversighters)
 * Implement first stages of wider socialization plan (central notice, landing page, tutorials)
 * Start implementing feedback page usability test plan (Pau, user testing.com)
 * Continue OmniTI training and documentation for hand-off in late June

All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Metrics
AFT v.5
 * For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
 * Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

AFT v.4
 * We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
 * Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
 * Global daily ratings and conversions
 * Volume of daily ratings per article
 * Most frequently rated articles
 * Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team
Here are the Wikimedia team members who are working on this project at this time:
 * VP Engineering / Product	Erik Moeller
 * Senior Product Manager	Howie Fung
 * Product Manager, Editor Engagement	Fabrice Florin (project lead and primary author of this page}
 * Features Engineering Director	Alolita Sharma
 * Senior Research Analyst	Dario Taraborelli
 * UX Designer	Brandon Harris
 * Code Review / Testing	Roan Kattouw
 * Community Outreach	Oliver Keyes
 * Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers Here are the team members from OmniTI, our development partner:
 * VP Business Development  Leon Fayer
 * Project Manager  Yoni Shostak
 * Developer  Reha Sterbin
 * Developer  Greg Chiasson
 * UI Developer/Designer  Sean Heavey

Community members Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:
 * Bensin
 * Dougweller
 * Looie496
 * RJHall
 * Utar

We’d like to give special thanks and recognition to these community members who have already done so much to develop this new tool: we are grateful for your insights and commitment to this cause!