Article feedback/Version 5

Option 1: Feedback - A new design for Version 5 of the Article Feedback Tool

This is the project overview page for Version 5 of the Article Feedback Tool. The Wikimedia Foundation is developing this new feature as an "on-ramp" to engage readers to contribute to Wikipedia -- and become editors over time.

Related pages:

• Feature Requirements
• Sample Articles for Testing
• Metrics Dashboard (first test results)
• Data and Metrics Plan
• Interactive prototype
• Technical Design
• Wikipedia article on AFT5
• Wikipedia talk page

Goals

Project slides for the article feedback tool version 5, including wireframes, flowcharts and plans, January 2, 2012

In October 2011, Wikimedia kicked off another round of product development on new and alternative methods of providing feedback regarding the quality of articles, including ideas like a moderated free-text comment queue for suggestions. The foundation has since hired Fabrice Florin as product manager for new editor engagement, as well as OmniTI in Maryland for the software development of these new features.

Our overall goals will be to 1) measure quality, 2) provide meaningful feedback to the editing community, 3) offer lightweight contribution tools as a gateway to editing. The idea is to try new experiments alongside the existing article feedback tool (i.e. to replace it on some reasonably representative subset articles with an alternative implementation, gather data, and iterate).

We invite the Wikipedia community (as well as all Wikimedians), to participate in this experiment. Together, we hope to create and test new collaborative tools towards these objectives:

• engage readers to participate more on Wikipedia
• give editors new tools to improve article quality
• encourage readers to become editors over time
• invite a collaboration between editors and readers
• experiment with outsourcing web development

The set of implementation of the Article Feedback Tool (Versions 1-4) were focused on the dual objectives of participation and quality. The existing tool intends to provide a quantitative measurement of quality of articles as well as an on-ramp for contribution (i.e., editing). Based on the research conducted by WMF, the tool shows promise in being an on-ramp for contribution. As implemented, the tool appears to provide a reasonable measurement for quality along some dimensions (Completeness and Trustworthy), while other dimensions of quality (Objective and Well-written) tend to show a lower correlation with ratings.

Based on this research and the input from the Wikipedia editing community (examples are here and here), the next version of the tool will focus even more heavily on participation. Editors told us that it would be valuable if they knew what readers were looking for. Version 5 of the Article Feedback Tool will focus on finding ways for readers to contribute productively to building the encyclopedia.

For example, we are developing a feedback form where we ask readers "Did you find what you were looking for?" We will also invite them to add a comment or suggestion for improvement. Even if this reader doesn't become an editor, the hope is that they will contribute productively by letting the editing community know what was missing from the article. As we did in the first few phases of the project, we will also invite them to make the edit themselves.

Throughout this project, we will continue to test various ways of measuring quality. We may, however, do this based on more implicit data. In the above example, the percentage of "yes"'s could be an indicator of article quality, even though we don't ask the reader to explicitly evaluate the quality of the article.

Features

Article Feedback Form Option 1

Article Feedback Form Option 2

Article Feedback Form Option 3

Article Feedback Call to Edit 1

Key features for AFT V5 include: (see also full feature requirements page)

• new feedback forms
• calls to action
• feedback page
• moderation tools
• expanded feedback

In the first phase of this project (Oct.-Dec. 2011), we developed and started testing three different feedback forms to extend the current rating tool:

• Option 1: Basic feedback
• Option 2: Make a suggestion
• Option 3: Review this page

We are now A/B testing these options to find out which is most effective for engaging readers, supporting editors and improving article quality (see data and metrics page).

In phases 1.2 to 1.5 (Jan.-Feb. 2012), we plan to test these additional options:

• Feedback links
• New calls to action
• A stand-alone call to edit
• Feedback page

In phase 2 (Feb.-March 2012), we plan to release some of these new features widely on Wikipedia, as well as develop more features, such as expanded feedback.

For a preview of what these forms and pages might look like, read below, or view our project slides, or check out this interactive prototype.

Get involved

The Wikimedia Foundation plans to involve the community at each step of the way, and is looking for editors and readers to contribute to the development of this article feedback tool.

Feedback Evaluation

The Foundation is now looking for experienced editors to help evaluate the quality of the feedback collected by the new AFT5 tools through the end of January 2012. To learn more about this project, visit the Feedback evaluation page. A special task force of editors is being trained for this evaluation and is actively reviewing feedback posts.

Request for comments

Please review the Request for Comments at the end of the talk page, regarding who should have the right to hide feedback collected by the Article Feedback Tool. This two-week Request for Comments will be open from January 6th to 20th, 2012.

Talk page

You're invited to give feedback about the Foundation's current plans for Version 5 and let them know what you think on the Talk page: what community concerns aren't being considered? Are there flaws in the current plans? How would you make the current ideas better? do you have any of your own to share? This is open to everyone - just drop your thoughts on the Talk page.

IRC chats

To invite community participation during the development of this tool, the Foundation also hosts frequent IRC chats during office hours, as outlined in the schedule below. We hope you will join us on future chats. In the meantime, here are logs of our earlier IRC chats on Oct. 27, Nov. 3, Nov. 10, Nov. 18 and Dec. 16, 2011 -- as well as Jan. 6, 2012.

Schedule

The first phase of Version 5's development is timetabled below; those segments marked in green are the ones where we will be asking for community feedback and participation. These dates are still tentative and subject to change, so please check this page often for schedule updates. Even if you cannot participate right away, there will be more opportunities to contribute in future stages of the project.

Stage	Date	Description
New Code Release (1.5a)	15 February 2012	Help test updated feedback page with new features.
Community Outreach	16 February 2012	An IRC chat with the community, asking for comments on phase 1.5 release.
New Code Release (1.5b)	22 February 2012	Help test updated feedback page and alternate feedback links.
Community Outreach	25 February 2012	An IRC chat with the community, asking for comments on new features.
Data Analysis Report, Phases 1.4 + 1.5	28 February 2012	Publish additional findings from phases 1.4 and 1.5, and analysis of edit survival
Wide Release of new AFT V5	7 March 2012	Release selected version of the feedback form and page on all pages of English Wikpedia
Design and Development, Phase 2	15 March -- through 30 June, 2012	Agile development of more features, bi-monthly code deliveries, based on community feedback

Weekly Update (Feb. 13)

Last week: (Feb. 6)

• Tested and fixed bugs on AFT5 feedback page (non-public, barebones)
• Designed Activity log, as well as Hide/Oversight panes, with mockups
• Discussed and refined Oversight workflows (Request, Decline, Un-oversight)
• Extended Tools panel with Request Oversight and Activity links
• Updated feature requirements page with new specs and mockups
• Hosted private IRC chat with oversighters to review and get buy-in for workflows
• Analyzed team survey and other data for selecting final feedback form
• Selected final feedback form (option 1: did you find way you were looking for?)
• Discussed and prepared new features for reducing noise on protected pages

This week's goals - Phase 1.5a: (Feb. 13)

• Develop new version of Advanced Feedback page with Activity & oversight features
• Deploy feedback page bug fixes and clicktracking patch on en-wki?
• Design/develop feedback button close box + user preference link
• Develop Other Feedback links test (option A, D or no link)
• Prepare Metrics test 2 (feedback link options)
• Prepare test plan and unit tests, start testing according to plan
• Finalize usability tests and prepare report
• Discuss feedback page features for reducing noise on protected pages
• Update help pages for feedback page
• Host private IRC chat with rollbackers workgroup to test feedback page

Next week's goals - Phase 1.5b: (Feb. 20)

• Pre-deploy Advanced Feedback page with Activity & oversight features (en-labs?)
• Release feedback button close box + user preference link?
• Release Other Feedback links test (option A, D w/ close box -- or no link)
• Monitor Metrics test 2 (feedback link options)
• Design feedback page noise-reduction features for protected pages
• Develop Abuse filter + Spam block, optimize performance
• Publish final reports for reader survey + team survey
• Publish final report and slides about 3 feedback forms
• Host IRC chat with regular users to test feedback page

Phase 1.6: (Feb. 27)

• Test Advanced Feedback page with Activity & oversight features (en-labs)
• Develop Abuse filter + Spam block, optimize performance
• Develop Option 4 (Edit this page CTA)
• Develop feedback page features for reducing noise on protected pages
• Design/Develop links to feedback page (in feedback forms and talk page)
• Design alternative versions of selected feedback form
• Prepare Metrics test 4 (alternative feedback forms)
• Write blog post and promote feedback page launch
• Host IRC chat with all members to discuss full AFT5 service

Phase 1.7a: (Mar. 5)

• Release Advanced Feedback page with Activity & oversight features (en-wiki)
• Release Option 4 (Edit this page CTA)
• Start Metrics test 3 (Edit CTA vs. Feedback forms)
• Test Advanced Feedback page with Abuse filters (public)

• Test links to feedback page (in feedback forms and talk page)
• Test solution for reducing noise on protected pages
• Develop alternative versions of selected feedback form
• Prepare usability tests and study results
• Host IRC chat with all members to discuss expanded AFT5 service

Phase 1.7b: (Mar. 12)

• Release Advanced Feedback page with Abuse filters (public)
• Release solution for reducing noise on protected pages
• Develop/Test alternative versions of selected feedback form
• Prepare Metrics test 4 (alternative feedback forms)
• Start feedback page usability tests and study results
• Host IRC chat with all members to discuss expanded AFT5 service

Phase 1.8a: (Mar. 19)

• Release alternative versions of selected feedback form
• Start Metrics test 4 (alternative feedback forms)
• Design 2 Additional CTAs (signup, email)
• Start usability tests for feedback form and study results
• More platform support (IE7, IOS?)
• Host IRC chat with all members to discuss expanded AFT5 service

Phase 1.8b: (Mar. 26)

• Test final feedback form and feedback page improvements TBD
• Develop 2 Additional CTAs (signup, email)
• Release links to feedback page (in feedback forms and talk page)
• Host IRC chat with all members to discuss expanded AFT5 service

Phase 1.9: (Apr. 2)

• Release final feedback form and feedback page improvements TBD
• Release 2 Additional CTAs (signup, email)
• Start wider deployment of AFT v5 feedback forms, links and page?

All dates above subject to change, based on agile development goals as well as Wikimedia's overall plans for this project.

Metrics

AFT v.5

• For live results of our current tests, check the metrics dashboard and feedback stream, as well as data dumps.
• Metrics and research questions that will be used to test AFT v.5, as well as a detailed plan with the breakdown of the different tests that we will run, can be found on this page.

Test plan for AFT v.5

AFT v.4

• We collected a number of high-level usage metrics (November 2011) from the current version of AFT as a baseline before starting the deployment of AFT v.5.
• Several dashboards with real-time data collected from AFT v.4 are available from the toolserver:
  • Global daily ratings and conversions
  • Volume of daily ratings per article
  • Most frequently rated articles
• Detailed reports from the analysis of data collected via AFT v.4 are available on this page.

Team

Here are the Wikimedia team members who are working on this project at this time:

• VP Engineering / Product Erik Moeller
• Senior Product Manager Howie Fung
• Product Manager, Editor Engagement Fabrice Florin (project lead and primary author of this page}
• Features Engineering Director Alolita Sharma
• Senior Research Analyst Dario Taraborelli
• UX Designer Brandon Harris
• Code Review / Testing Roan Kattouw
• Community Outreach Oliver Keyes
• Research Consultant Aaron Halfaker

(note that Oliver and Aaron are Wikimedia contractors -- all others are WMF employees)

Developers
Here are the team members from OmniTI, our development partner:

• VP Business Development Leon Fayer
• Project Manager Yoni Shostak
• Developer Reha Sterbin
• Developer Greg Chiasson
• UI Developer/Designer Sean Heavey

Community members
Here are some of the Wikipedia community members who have joined our workgroup to help design this product together:

• Bensin
• Dougweller
• Looie496
• RJHall
• Utar