Article feedback/Version 5

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:
 * Feature Requirements
 * Sample Articles for Testing
 * Metrics Dashboard (first test results)
 * Data and Metrics Plan
 * Interactive prototype
 * Technical Design
 * Wikipedia article on AFT5
 * Wikipedia talk page

Goals
In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. The foundation has since hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

Our overall goals will be to 1) measure quality, 2) provide meaningful feedback to the editing community, 3) offer lightweight contribution tools as a gateway to editing. The idea is to try new experiments alongside the existing article feedback tool (i.e. to replace it on some reasonably representative subset articles with an alternative implementation, gather data, and iterate).

We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:
 * engage readers to participate more on Wikipedia
 * give editors new tools to improve article quality
 * encourage readers to become editors over time
 * invite a collaboration between editors and readers
 * experiment with outsourcing web development

The set of implementation of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. The existing tool intends to provide a quantitative measurement of quality of articles as well as an on-ramp for contribution (i.e., editing). Based on the research conducted by WMF, the tool shows promise in being an on-ramp for contribution. As implemented, the tool appears to provide a reasonable measurement for quality along some dimensions (Completeness and Trustworthy), while other dimensions of quality (Objective and Well-written) tend to show a lower correlation with ratings.

Based on this research and the input from the Wikipedia editing community (examples are here and here), the next version of the tool will focus even more heavily on participation. Editors told us that it would be valuable if they knew what readers were looking for. Version 5 of the Article Feedback Tool will focus on finding ways for readers to contribute productively to building the encyclopedia.

For example, we are developing a feedback form where we ask readers "Did you find what you were looking for?" We will also invite them to add a comment or suggestion for improvement. Even if this reader doesn't become an editor, the hope is that they will contribute productively by letting the editing community know what was missing from the article. As we did in the first few phases of the project, we will also invite them to make the edit themselves.

Throughout this project, we will continue to test various ways of measuring quality. We may, however, do this based on more implicit data. In the above example, the percentage of "yes"'s could be an indicator of article quality, even though we don't ask the reader to explicitly evaluate the quality of the article.

Features


Key features for AFT V5 include: (see also full feature requirements page)
 * new feedback forms
 * calls to action
 * feedback page
 * moderation tools
 * expanded feedback

In the first phase of this project (Oct.-Dec. 2011), we developed and started testing three different feedback forms to extend the current rating tool: We are now A/B testing these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see data and metrics page).
 * Option 1: Basic feedback
 * Option 2: Make a suggestion
 * Option 3: Review this page

In phases 1.2 to 1.5 (Jan.-Feb. 2012), we plan to test these additional options: In phase 2 (Feb.-March 2012), we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.
 * Feedback links
 * New calls to action
 * A stand-alone call to edit
 * Feedback page

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this.

Get involved
The Wikimedia Foundation plans to involve the community at each step of the way, and is looking for editors and readers to contribute to the development of this article feedback tool.

Feedback Evaluation
The Foundation is now looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools through the end of January 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained for this evaluation and is actively reviewing feedback posts.

Request for comments
Please review the Request for Comments at the end of the talk page, regarding who should have the right to hide feedback collected by the Article Feedback Tool. This two-week Request for Comments will be open from January 6th to 20th, 2012.

Talk page
You're invited to give feedback about the Foundation's current plans for Version 5 and let them know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats
To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule below. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- as well as Jan. 6, 2012.

Schedule
The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project. All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Weekly Update
Last week: (Feb. 27) This week: (Mar. 5) All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.
 * Tested and filed bugs for next version of Advanced Feedback page (Activity/oversight)
 * Tested AFT5 on new MediaWiki 1.19 platform (on prototype)
 * Researched and designed feedback button “X” close box with Heather
 * Designed Option E Feedback links test (option A, E or no link)
 * Prepared data requirements and clicktracking specs for Metrics test 2 (feedback link options)
 * Discussed designs for feedback page quality filters / noise-reduction on controversial pages
 * Updated schedule and timeline for AFT5 in Q1 2012, based on team recommendations
 * Prepared unit tests and Selenium tests, developed overall testing plan
 * Integrated analysis of Option 1 sub-options (Aaron, Dario)
 * Deploy Advanced Feedback page on March 7 at 10am PT
 * Update help pages for feedback page to document new Activity/oversight features
 * Design/Develop improvements to oversight process on feedback page, as needed
 * Develop feedback button “X” close box + user preference link
 * Develop Other Feedback links test (option A, E or no link)
 * Finalize Metrics test 2 (feedback link options) for March 21 deployment
 * Develop Abuse filter + Spam block, optimize performance
 * Design new feedback page features for quality filters, editor engagement and noise reduction
 * Host IRC chat with workgroup to discuss full AFT5 feedback page
 * Review/edit final reports for reader survey + team survey
 * Review/edit final report and slides about 3 feedback forms

Metrics
AFT v.5
 * For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
 * Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

AFT v.4
 * We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
 * Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
 * Global daily ratings and conversions
 * Volume of daily ratings per article
 * Most frequently rated articles
 * Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team
Here are the Wikimedia team members who are working on this project at this time:
 * VP Engineering / Product	Erik Moeller
 * Senior Product Manager	Howie Fung
 * Product Manager, Editor Engagement	Fabrice Florin (project lead and primary author of this page}
 * Features Engineering Director	Alolita Sharma
 * Senior Research Analyst	Dario Taraborelli
 * UX Designer	Brandon Harris
 * Code Review / Testing	Roan Kattouw
 * Community Outreach	Oliver Keyes
 * Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers Here are the team members from OmniTI, our development partner:
 * VP Business Development  Leon Fayer
 * Project Manager  Yoni Shostak
 * Developer  Reha Sterbin
 * Developer  Greg Chiasson
 * UI Developer/Designer  Sean Heavey

Community members Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:
 * Bensin
 * Dougweller
 * Looie496
 * RJHall
 * Utar