Article feedback/status

Last update on: 2012-07-monthly

2010-09-01
We’re in the beginning phase of this project, figuring out requirements and generally determining the scope of our near-term and long-term efforts in this area. We are currently working on a pilot rating system which will be available as part of the Public Policy pilot program in late September.

2010-10-01
Phase 1, deployed last week in a pilot experiment for the Public Policy Initiative. The goal is to gather metrics and learn about how this feature should work. We plan to use this information to help develop requirements for the next phase in the first quarter of 2011.

2010-11-01
The first version of this feature was rolled out at the end of September, and we spent the month of October collecting and analyzing the feedback from the tool. We are currently designing the second version of this feature, and starting development later this month. We plan to roll out the next phase in sync with the Public Policy Initiative Phase 2 in January.

2010-12-01
In development. The initial version (phase 1) has been in production since the end of September on English Wikipedia. Phase 2, focusing on usability improvements, performance improvements and wider deployment, is slated for release in January 2011.

2011-01-01
In development. The initial version (phase 1) has been in production since the end of September on English Wikipedia. Phase 2, slated for release in January 2011, has been under development. Planning for phase 3 will begin soon.

2011-02-01
Phase 2 is currently in development. The requirements have validated against the current codebase and UI. This feature will be dark launched on prototype on February 4. We'll wait until after the 1.17 deployment before making this feature live due to Resource Loader dependencies.

2011-03-01
The deployment to our prototype has surfaced additional feature requirements that we've now addressed. Now that MediaWiki 1.17 has been successfully deployed, we can release the latest version of the Article feedback tool on the English Wikipedia, as part of our pilot experiment this quarter. Requirements for the next version (3.0) are being drafted.

2011-04-01
The second phase of this feature was released on the English Wikipedia in mid March. A major change in the interface is the ability for reviewers to specify the source of their knowledge, e.g. if they have an academic degree in a related field (see screenshot). Experiments to encourage user engagement are being performed as well. Dario Taraborelli also published an analysis of the first phase experiment. We're currently expanding the scope of the experiment to include several thousand articles, in order to get results that are more meaningful statistically.

2011-05-01
<section begin=2011-05-01 /> <section end=2011-05-01 />
 * Phase 2: Trevor Parscal implemented the expiration of ratings, added error handling mechanisms and fixed IE bugs. Roan Kattouw reviewed and deployed the code to production.
 * Phase 3: Timo Tijhof and Trevor Parscal implemented the EmailCapture extension, a tool that allows unregistered users rating an article to leave their e-mail address if they want to be contacted later by the Community department. Weekly deployments were performed by Roan Kattouw. The team started to work on the dashboard, a summary page to surface general rating trends.

2011-06-01
<section begin=2011-06-01 />Version 3 was deployed to the English Wikipedia on May 9 with new features, like the article feedback dashboard, a summary page showing general rating trends. The experiment was expanded to 100,000 articles and may be expanded further after analysis of the results. The next version of the Article Feedback feature is currently in development.<section end=2011-06-01 />

2011-07-01
<section begin=2011-07-01 />Additional features were added in June, like a dashboard tracking articles receiving low ratings. Roan Kattouw started to implement the UDP back-end to provide clicktracking metrics to assess user engagement. The community provided feedback and bug reports, and the development team addressed the concerns raised, for example by implementing a user preference to hide the tool. Tooltips were also added to provide more information on the meaning of the star ratings. Dario Taraborelli continued to evaluate the data provided by the articles already showing the feature. The incremental roll-out to all articles on the English Wikipedia is planned to be completed by mid-July.<section end=2011-07-01 />

2011-08-01
<section begin=2011-08-01 />Roan Kattouw completed the UDP logger (for clicktracking metrics) and deployed it to production. The Article feedback feature was incrementally rolled out to all articles on the English Wikipedia, and the Product research team continued to analyze its impact (read more).<section end=2011-08-01 />

2011-08-31
<section begin=2011-08-31 />Development was completed in July; August was mostly devoted to data analysis. Dario Taraborelli analyzed the volume of edits, and couldn't yet find any statistically significant difference in edits before and after the activation of the feature on English Wikipedia articles. In order to clarify licensing and privacy policies regarding the data (and to facilitate its reuse by external tools & researchers), an explanation was published, stating that user feedback data (from Article feedback and MoodBar, for example) were considered public contributions just like any edit.<section end=2011-08-31 />

2011-09-30
<section begin=2011-09-30 />Dario Taraborelli continued to analyze the data recorded. Howie Fung assessed proposals from vendors in response to the RfP to build an extended review system, that would notably integrate free-text comments to the Article feedback feature.<section end=2011-09-30 />

2011-10-03
<section begin=2011-10-03 />Planning and design work begins on a new version 5 of the Article Feedback Tool. Fabrice Florin is engaged as product consultant to lead the development of AFT V5.<section end=2011-10-03 />

2011-10-31
<section begin=2011-10-31 />Work has started on Version 5 of this feature, which will focus on reader engagement, for example by offering new feedback forms and free-text comment fields; a comment moderation system is also planned. This project is led by contractor Fabrice Florin, and will be implemented by an external contracting company, OmniTI.<section end=2011-10-31 />

2011-11-30
<section begin=2011-11-30 />Fabrice Florin completed feature requirements, mockups and project plan for Version 5, and led development of first beta code by OmniTI. Dario Taraborelli started to work on the version 5 data model, metrics requirements and data analysis plan. He also worked with Fabrice and Howie Fung on guidelines and requirements for marking of AFT-generated edits and account tracking. A series of real-time dashboards for AFT data were uploaded to the Toolserver. Roan Kattouw reviewed the proposed database schema, and showed the OmniTI team how to deploy code to the prototype server.<section end=2011-11-30 />

2011-12-31
<section begin=2011-12-31/>Aaron Halfaker, Oliver Keyes and Dario Taraborelli have finished gathering valuable data from the community about the usefulness of comments coming in from each of the three forms launched in December. A survey to get comments from readers about the effectiveness and attractiveness of each design was also introduced, and the team has been compiling the various sets of data to produce a report on the pros and cons of each form. Fabrice Florin is leading development on the next round of features, including a new feedback page, to be implemented by OmniTI, our development partner. <section end=2011-12-31/>

2012-01-31
<section begin=2012-01-31/>Fabrice Florin led development on the next round of Article Feedback Tool v5 features, including a new feedback page, to be released in early February by OmniTI, our development partner. Aaron Halfaker, Oliver Keyes and Dario Taraborelli continued to collect valuable data from the community about the usefulness of comments coming in from each of the three forms launched in December. A survey to get comments from readers about the effectiveness and attractiveness of each design was also launched, and the team has been compiling the various sets of data to produce a report on the pros and cons of each forms in early February. The target date for an expanded feedback page is Feb. 15 for pre-deployment testing on en-labs, then wider deployment on Feb. 22.<section end=2012-01-31/>

2012-02-29
<section begin=2012-02-29/>Fabrice Florin is working with our development partner OmniTi on version 5 of this tool, with the help of community liaison Oliver Keyes and analyst Dario Taraborelli. This month, the team created a new feedback page, with special features to be tested with oversighters and rollbackers in early March. Roan Kattouw has completed code review for compatibility with 1.19 and will deploy a new release on March 8, as well as a new working test environment in coming weeks. Final reports for phase 1 of this project will be published in early March. <section end=2012-02-29/>

2012-03-31
<section begin=2012-03-31/>Fabrice Florin worked with OmniTi to develop new features for version 5 of the Article Feedback Tool (AFT5). This month, the team created new feedback links, new monitoring tools for editors and for oversighters, and started development on an abuse filter and a relevance filter. Brandon Harris and Heather Walls helped enhance the article feedback design, creating new icons for the monitoring tools. The AFT5 team also published its first report on phase 1 of AFT5, co-authored by Fabrice and Dario Taraborelli, with Howie Fung, Oliver Keyes and Aaron Halfaker. Roan Kattouw helped solve some tricky technical issues and deployed several new releases with the team this month. Current goals for this project are to complete feature development by the end of May, with full deployment in the summer. <section end=2012-03-31/>

2012-04-monthly
<section begin=2012-04-monthly/>Fabrice Florin worked with OmniTi to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team deployed the first versions of the abuse filter and the relevance filter, as well as new monitoring tools for editors, to help surface useful suggestions and reduce the noise on the feedback page where posts are listed for each article. Pau Giner started designing a new look and feel for that feedback page, as well as a final version of the feedback form, with guidance from Brandon Harris. We also finalized a set of special tools for oversighters, as well as new feedback links. Dario Taraborelli, Oliver Keyes and Aaron Halfaker collected and analyzed data on how prominent feedback links impact both volume and quality of user feedback. Roan Kattouw continued to review our code and help deploy weekly releases, and Matthias Mullie contributed new code for this project. We expect to complete feature development by the end of May, with full deployment in the summer.<section end=2012-04-monthly/>

2012-05-monthly
<section begin="2012-05-monthly"/>Fabrice Florin worked with OmniTI and new WMF engineer Matthias Mullie to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team deployed a new look and feel for the article feedback page (based on a streamlined design by Pau Giner) as well as a central feedback page, where editors can monitor posts from all articles on Wikipedia. We also developed a [//en.wikipedia.org/wiki/San_Francisco?aftv5_form=6 final feedback form] (scroll to bottom of page), which gradually engages users to contribute to the encyclopedia. Dario Taraborelli, Oliver Keyes and Aaron Halfaker collected and analyzed data on how posting feedback impacts both conversion and newcomer quality. Based on this analysis, we now project over 2 million feedback posts per month on the English Encyclopedia when the tool is widely deployed later this year (on par with the total number of edits per month). Our research suggests that posting feedback encourages a substantial number of users to productively edit articles on Wikipedia, which is expected to help reverse the recent decline in both new and existing editors. Roan Kattouw continued to review our code and deploy weekly releases, while training our team to deploy their own code over time. We are planning for a wider deployment by the end of June, with full deployment a couple months later.<section end="2012-05-monthly"/>

2012-06-18
<section begin="2012-06-18"/>Last week: (June 11)
 * Developed and deployed a range of features and bug fixes based on community/testing feedback
 * Started development on remaining features (CTA4, My Last Post, My Contributions, etc.)
 * Discussed plan for increasing article sample size for wider deployment
 * Wrapped up Metrics Stage 5 clicktracking (Option 1/CTA3 vs. Option 6/CTA5)
 * Removed CTA3 linking to new survey 6 for feedback form (Option 6 / CTA 3)
 * Tweaked final feedback form design (Option 6, progressive reveal)
 * Tweaked talk page link to be closer to title
 * Discussed adding Moodbar reply feature to AFT with Benny + Matthias for later this summer
 * Edited and published help FAQ pages for all user groups (inc. monitors, oversighters)
 * Discussed wider socialization plan first draft (Oliver)
 * Continued OmniTI training and documentation for hand-off in late June
 * Made final release decisions in project review with Erik and Howie

This week: (June 18)
 * Test and deploy CTA 4 (Sign up or login)
 * Test and deploy My Last Post (show it on top, with New label)
 * Test and deploy My Contributions (test on prototype first?)
 * Develop, test (and deploy?) for increasing article sample size for wider deployment
 * Test and deploy updated feedback form design (Option 6, progressive reveal)
 * Develop and test IE8 fixes and other configuration issues
 * Develop and test more features and bug fixes based on community/testing feedback
 * Test and deploy default warning message for Abuse filter (Andrew)
 * Tweak talk page link to be closer to title - hold release until July 3
 * Prioritize Bugzilla bug list with development team members
 * Update help FAQ pages for all user groups (inc. monitors, oversighters)
 * Implement first stages of wider socialization plan (central notice, landing page, tutorials, screencast)
 * Start implementing feedback page usability test plan (Pau, user testing.com)
 * Continue OmniTI training and documentation for hand-off in late June<section end="2012-06-18"/>

2012-07-02
<section begin="2012-07-02"/>Last week: (June 25)
 * Tested lottery code to increase article sample size for wider deployment
 * Tested IE8 fixes and other configuration issues
 * Tested feedback on My Contributions (includes changes to core)
 * Tested Suppression Log formatting update
 * Tested default warning message for Abuse filter + new abuse filters
 * Developed and tested final bucketing for Calls to Action (CTA1, CTA2, CTA 4)
 * Developed and tested changes to editor's view (show 'View activity' and other tools)
 * Tweaked My Last Post (use different label, remove if hidden, separate from toolbar)
 * Developed and test more features and bug fixes based on community/testing feedback
 * Started wider socialization plan and schedule (central notice, landing page, tutorials)
 * Reviewed copy and design, then develop and edit landing page and central notice banner
 * Completed and started reviewing tutorial writing and design
 * Recorded and edited first draft of screencast video (short 2 min. version)
 * Published blog post about Metrics Stage 3 results (conversion / productivity)
 * Prepared to wrap up Metrics Stage 4 study and data collection for feedback page
 * Completed OmniTI documentation and had final hand-off meeting at end of week
 * Worked with Chris and Jeremy on unit testing and OmniTI knowledge transfer
 * Continued scalability discussion with key members of engineering team

This week: (July 2) <section end="2012-07-02"/>
 * Test and deploy lottery code to increase article sample size for wider deployment
 * Test and deploy changes to editor's view (show 'View activity' and other tools)
 * Test and deploy final bucketing for Calls to Action (CTA1, CTA2, CTA 4)
 * Test and deploy IE8 fixes and other configuration issues
 * Test and deploy feedback on My Contributions (includes changes to core)
 * Test and deploy Suppression Log formatting update
 * Test and deploy default warning message for Abuse filter + new abuse filters
 * Test and deploy My Last Post (use different label, remove if hidden, separate from toolbar)
 * Develop and test more features and bug fixes based on community/testing feedback
 * Continue to prepare wider socialization plan and schedule (central notice, landing page, tutorials)
 * Finalize copy and design, then develop and edit landing page and central notice banner
 * Complete tutorial writing, design and editing - by end of week
 * Finalize copy and record screencast video, edit and post short 2 min. version
 * Prepare short blog post about wider deployment (for publication next week)
 * Wrap up Metrics Stage 4 study and data collection for feedback page
 * Work with Chris and Jeremy on unit testing and quality assurance
 * Continue scalability discussion with key members of engineering team

2012-06-monthly
<section begin="2012-06-monthly"/>Fabrice Florin worked with new WMF engineer Matthias Mullie and OmniTI to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team completed primary feature development for this tool, including the article feedback page, the central feedback page, and the [//en.wikipedia.org/wiki/San_Francisco?aftv5_form=6 final feedback form] (scroll to bottom of page). We started writing and publishing new documentation about this project, including this help page. Dario Taraborelli, Aaron Halfaker and Oliver Keyes published a full report that suggests that people who post feedback are more likely to edit articles afterwards. Roan Kattouw continued to review our code and trained our team to start deploying code on their own. We have started a wider deployment of AFT, which will gradually increase our coverage to 10% of the English encyclopedia by the end of July, with full deployment a couple months later.<section end="2012-06-monthly"/>

2012-07-monthly
<section begin="2012-07-monthly"/>Fabrice Florin worked with Matthias Mullie to deploy Article Feedback on 10% of the English Encyclopedia, with the help of Pau Giner, Roan Kattouw, Oliver Keyes and Chris McMahon. This month, the team developed final features for this tool, included on the article feedback page, the central feedback page, and the [//en.wikipedia.org/wiki/San_Francisco?aftv5_form=6 final feedback form] (scroll to bottom of page). We have received a very positive response to article feedback from the Wikipedia community through a variety of channels, from Wikimania presentations to IRC chats, talk pages and surveys (which consistently show a 70% satisfaction level from both readers and editors). Community members typically find the tool useful and well-thought out, and many editors have told us they have already made improvements to articles based on feedback from readers -- which is exactly the behavior we were hoping to encourage! Ryan Kaldari and Benny Situ helped review our code and provided valuable advice. We have started our productization phase (more platforms, scaleability, code re-factoring, localization, metrics, mobile). We are now aiming for a full release to 100% of English Wikipedia by October 2012 -- with other wiki projects later this year. <section end="2012-07-monthly"/>