Article feedback/Version 5

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:
 * Feature Requirements
 * Sample Articles for Testing
 * Metrics Dashboard (first test results)
 * Data and Metrics Plan
 * Interactive prototype
 * Technical Design
 * Wikipedia article on AFT5
 * Wikipedia talk page

Goals
In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. The foundation has since hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

Our overall goals will be to 1) measure quality, 2) provide meaningful feedback to the editing community, 3) offer lightweight contribution tools as a gateway to editing. The idea is to try new experiments alongside the existing article feedback tool (i.e. to replace it on some reasonably representative subset articles with an alternative implementation, gather data, and iterate).

We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:
 * engage readers to participate more on Wikipedia
 * give editors new tools to improve article quality
 * encourage readers to become editors over time
 * invite a collaboration between editors and readers
 * experiment with outsourcing web development

The set of implementation of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. The existing tool intends to provide a quantitative measurement of quality of articles as well as an on-ramp for contribution (i.e., editing). Based on the research conducted by WMF, the tool shows promise in being an on-ramp for contribution. As implemented, the tool appears to provide a reasonable measurement for quality along some dimensions (Completeness and Trustworthy), while other dimensions of quality (Objective and Well-written) tend to show a lower correlation with ratings.

Based on this research and the input from the Wikipedia editing community (examples are here and here), the next version of the tool will focus even more heavily on participation. Editors told us that it would be valuable if they knew what readers were looking for. Version 5 of the Article Feedback Tool will focus on finding ways for readers to contribute productively to building the encyclopedia.

For example, we are developing a feedback form where we ask readers "Did you find what you were looking for?" We will also invite them to add a comment or suggestion for improvement. Even if this reader doesn't become an editor, the hope is that they will contribute productively by letting the editing community know what was missing from the article. As we did in the first few phases of the project, we will also invite them to make the edit themselves.

Throughout this project, we will continue to test various ways of measuring quality. We may, however, do this based on more implicit data. In the above example, the percentage of "yes"'s could be an indicator of article quality, even though we don't ask the reader to explicitly evaluate the quality of the article.

Features


Key features for AFT V5 include: (see also full feature requirements page)
 * new feedback forms
 * calls to action
 * feedback page
 * moderation tools
 * expanded feedback

In the first phase of this project (Oct.-Dec. 2011), we developed and started testing three different feedback forms to extend the current rating tool: We are now A/B testing these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see data and metrics page).
 * Option 1: Basic feedback
 * Option 2: Make a suggestion
 * Option 3: Review this page

In phases 1.2 to 1.5 (Jan.-Feb. 2012), we plan to test these additional options: In phase 2 (Feb.-March 2012), we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.
 * Feedback links
 * New calls to action
 * A stand-alone call to edit
 * Feedback page

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this.

Get involved
We are developing this new tool in collaboration with Wikipedia editors, with whom we meet regularly over IRC and other channels. We are looking for more volunteers to help improve this article feedback tool.

Talk page
You're invited to give feedback about the Foundation's current plans for Version 5 and let us know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats
To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule on this page. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- as well as Jan. 6, 2012.

Feedback Evaluation
The Foundation is looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools in 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained to review feedback posts for this evaluation.

Schedule
The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project. All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Weekly Update
Last week: (Mar. 12) This week: (Mar. 19) All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.
 * Deployed clicktracking changes to track article names (+ AFT4 CTA changes)
 * Developed feedback button “X” close box + user preference link
 * Developed Other Feedback links test (option A, E or no link)
 * Developed/discussed improvements to oversight on feedback page and central activity log
 * Designed new feedback page features (Feature this post, Mark as Resolved, Relevance Filter)
 * Continued unit tests and Selenium tests, developed overall testing plan (Yoni)
 * Started to write blog post about phase 1 results
 * Release feedback button “X” close box + user preference link
 * Release Other Feedback links test (option A, E or no link)
 * Release improvements to oversight process on feedback page and central activity log
 * Develop new feedback page features (Feature this post, Mark as Resolved, Relevance Filter)
 * Develop Abuse filter + Spam block, optimize performance
 * Discuss/design improvements for monitors/oversighters on feedback page and activity logs
 * Start design of alternative versions of selected feedback form
 * Edit and publish blog post about phase 1
 * Review/edit final report and slides about 3 feedback forms
 * Review/edit final reports for reader survey + team survey
 * Develop Option 4 (Edit this page CTA)
 * Prepare Metrics test 3 (Edit CTA vs. Feedback forms)

Metrics
AFT v.5
 * For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
 * Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

AFT v.4
 * We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
 * Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
 * Global daily ratings and conversions
 * Volume of daily ratings per article
 * Most frequently rated articles
 * Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team
Here are the Wikimedia team members who are working on this project at this time:
 * VP Engineering / Product	Erik Moeller
 * Senior Product Manager	Howie Fung
 * Product Manager, Editor Engagement	Fabrice Florin (project lead and primary author of this page}
 * Features Engineering Director	Alolita Sharma
 * Senior Research Analyst	Dario Taraborelli
 * UX Designer	Brandon Harris
 * Code Review / Testing	Roan Kattouw
 * Community Outreach	Oliver Keyes
 * Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers Here are the team members from OmniTI, our development partner:
 * VP Business Development  Leon Fayer
 * Project Manager  Yoni Shostak
 * Developer  Reha Sterbin
 * Developer  Greg Chiasson
 * UI Developer/Designer  Sean Heavey

Community members Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:
 * Bensin
 * Dougweller
 * Looie496
 * RJHall
 * Utar

We’d like to give special thanks and recognition to these community members who have already done so much to develop this new tool: we are grateful for your insights and commitment to this cause!