Editor engagement experiments/status

Last update on: 2013-04-monthly

2012-04-monthly

Karyn Gladstone, Steven Walling, Maryana Pinchuk and Ryan Faulkner conducted the Necromancy experiment, emailing lapsed editors to encourage them to edit Wikipedia again. Work on the Template A/B testing project is wrapping up; a full report is expected in May. The E3 team will also be publishing details on each experiment on meta and the English Wikipedia. The technical specifications for each implementation will be posted on mediawiki.org. The team also began recruiting for its open positions; the first software engineer for the team will be joining mid May.

2012-05-monthly

The team started the development of the Timestamp Position Modification experimental feature, which was deployed then disabled due to a conflict between the ClickTracking feature and the MediaWiki API. Further testing and tuning continues, as well as analysis, redesign and development of the ClickTracking extension. We are gathering requirements for the next experiment on analyzing post-edit feedback, and we continue to hire software engineers.

2012-06-monthly

The team redeployed the Timestamp Position Modification experiment and it is now wrapped and in analysis. Designs and analytics work on the next experiment, post-edit feedback, were completed in preparation for a July deployment. Debug hooks were added to the clicktracking extension with the goal of improving QA for experiments. We wrote a clicktracking dashboard that intercepts event logging calls and displays them on-screen, shows which experiments are currently active, and to which bucket (if any) the current user has been assigned. Work is ongoing on a re-write of the clicktracking extension, which is taking shape as at Extension:E3_Experiments.

2012-07-monthly

The Timestamp Position Modification experiment was completed, and initial analysis shows that adding the timestamp on articles increases clicks on the History tab. Development started to deliver post-edit feedback messages; this experiment includes a proof-of-concept dry run of a new editor bucketing strategy for delivering experimental treatments that was deployed in advance of the full experiment. The team configured a test environment on Labs, to be used for UI and functional requirements validation by the team. The Wikimania conference was an opportunity to interact with editors from the English Wikipedia, and to define a new experiment related to cleanup templates.

2012-08-monthly

We deployed and ran the first iteration of post-edit feedback, testing whether various types of positive feedback after submission of an edit increase the productivity and retention of Wikipedia editors. (The results will be publicized soon.) We are currently working on the next iteration of post-edit feedback and on a new experiment which centers around the account creation process. We've also deployed click-tracking to the English Wikipedia community portal, account creation page, and the article edit form, and devised a tool for generating reports from the raw log data. Working with Asher Feldman, we've also architected an alternative data pipeline for event tracking, and begun its deployment.

2012-09-monthly

This month the E3 team announced the results of the first iteration of the post-edit feedback experiment, and worked on productization of the most successful confirmation message in a new extension, as well as through collaboration with the VisualEditor team. In addition, the team deployed the second iteration of experimental post-edit feedback, which lets new editors know when they reach important editing milestones early in their participation on Wikipedia. E3 also continued readying work on account creation user experience and the new event logging and usertagging analytics infrastructure to support feature experimentation, all of which are in alpha deployments to English Wikipedia.

2012-10-monthly

In October, the E3 team permanently deployed a confirmation message for all editors (Extension:PostEdit) to 16 Wikipedias, including six of the top 10 projects by size, and worked on associated maintenance of the feature. The team also deployed two iterations of tests for a new registration page (read more), including the beginnings of an API for client-side validation of the sign-up form. In support of current and future work, we deployed the beta EventLogging extension, a new architecture to replace the older ClickTracking extension. Last but not least, work started on redesigning the login and new experiments aimed at onboarding new Wikipedians.

2012-11-monthly

In November, the Editor Engagement Experiments team (E3) deployed the third and final A/B test of the new account creation page, including client-side validation. Results from basic data analysis of all three tests were published on Meta, and the project will now move to the productization stage. Extension:PostEdit was put in maintenance mode after being deployed to a further seven Wikipedias, including French and Portuguese. On the analytics side, E3 transitioned permanently to Extension:EventLogging for data collection purposes, and collaborated with the mobile team to track activity on Wikipedia's mobile beta. Last but not least, the team also deployed a small design improvement to the personal tools menu in MediaWiki core, in collaboration with the Language Engineering team.

2012-12-monthly

In December, the Editor Engagement Experiments team launched a new test aimed at Onboarding new Wikipedians. This interface delivers an optimized task list immediately after sign up, inviting those without an idea of how to get started to choose an article and try their hand at editing. The related GettingStarted extension was deployed mid-month and continued to evolve throughout the month, as early quantitative and qualitative research was conducted.

To go along with the launch of GettingStarted and other experimentation, EventLogging underwent heavy development, including the launch of a new Schema namespace on Meta for defining the data collected in a public, collaborative manner. We created production schemas for GettingStarted, account creation, mobile, and more. Ori Livneh also reworked the format, transmission, and cleanliness of data delivered to analysts and product managers, automatically generating database tables from these schemas for incoming events.

Late in the month, the team collaborated with fundraising to reach out to donors and readers as part of the annual fundraising campaign via email and a "Thank You" banner which ran at the end of the year. In addition to introducing millions of donors and readers to the Wikipedia editor community and inviting them to join, this campaign helped the team establish an experimental baseline for what a campaign to convert readers might look like.

In addition to the above launches, we continued development of the new account creation experience and Guided Tours by Matt Flaschen, which will be launched in January 2013. Active development was also begun by Ryan Faulkner and Dario Taraborelli on a user metrics API. The effort is threefold: to standardize user metrics in data analysis, to build infrastructure to efficiently compute metrics for a large set of users, and finally to expose those results via an API.

2013-01-monthly

In January, the Editor Engagement Experiments team ("E3") planned its goals for the quarter, which ends in March. We also made progress on the following projects which are included in that plan.

First up, we launched guided tours on the English Wikipedia, including a test tour to demonstrate the capabilities of the extension, and a tour associated with the "onboarding new Wikipedians" (aka GettingStarted) project. In addition to tours created by the team, the extension supports community-created tours. Note that unlike many other projects by the E3 team, guided tours are planned as a permanent addition to Wikipedia, with each tour implementation considered to be experimental. (For example: the "getting started" tour will be delivered via a split A/B test.)

While building guided tours, the team also A/B tested the Getting Started landing page and task list, measuring the effect it had on driving new contributions. Several rounds of analysis were completed and published on Meta (round 1, round 2), with the conclusion that the onboarding experience is leading to small but statistically significant increases in new English Wikipedians attempting to edit, as well as saving their first edit. In addition to measuring the effects of the guided tour associated with this project, immediate plans are to redesign the landing page and add additional task types, to entice more new contributors.

Work also continued on refining the reliability and precision of the data collected from EventLogging. In particular, we migrated EventLogging to a dedicated database, and began collecting server-side events in addition to client-side, to support work such as measuring account creations on desktop and mobile. January also saw the heavy use of the new User Metrics API, in order to complete cohort analysis of onboarding users and for metrics reported at the Board presentation on the Foundation's year-to-date progress. Development of the API continues, and a public announcement is expected for early March. Last but not least, a call was put out for a part-time Technical Writer to work on documenting both of these pieces of infrastructure.

2013-02-monthly

In February, the Editor Engagement Experiments team ("E3") continued working toward completing its goals for the quarter ending in March, which includes updates on the following projects.

After the intial launch of guided tours, Matt Flaschen and other team members worked on A/B testing the effectiveness of guided tours as part of the onboarding new Wikipedians experiences currently enabled on English Wikipedia. Results from these controlled tests are vital to understanding the impact of tours on editor engagement. In the meantime, the GuidedTour extension was enabled on Wikimedia Commons and six Wikipedias (including French, German, and Dutch), so that local administrators and volunteer developers could take advantage of the feature.

In addition to working on polishing and quantifying the effect of guided tours, significant progress was made on a new landing page for the onboarding project, with plans to launch early in March. The new Getting Started page will be expanded to include a wider variety task types offered to new editors. It will also be generated from a basic recommender system coupled with the GettingStarted extension, rather than relying on a bot.

Kirsten Menger-Anderson joined the team as Technical Writer mid-month. She began work with Ori Livneh, Dario Taraborelli, and others on documenting the EventLogging extension, with the goal of producing a comprehensive guide for end users of EventLogging, especially other Wikimedia Engineering teams in need of data. Future work by Kirsten will include similar documentation of the User Metrics data analysis API, which will be opened up for internal use in March.

2013-03-monthly

In March, the Editor Engagement Experiments team largely placed other projects – such as guided tours, EventLogging, and others – on hold to focus on two key initiatives: the "Getting Started" process for onboarding new Wikipedians, and on making the redesign of account creation and login a permanent, internationalized part of MediaWiki core.

For the Getting Started project, the team launched a new version on English Wikipedia, which included a new landing page with additional types of tasks suggested for brand new editors to try. The list of tasks is now generated by a basic recommender system built by Ori Livneh, which gathers, filters, and delivered a fresh list of tasks automatically for every editor. This new backend paves the way for releasing the "getting started" feature on other projects, after we've completed data analysis and testing to understand which kinds of tasks are ideal for first time editors. Additionally, Matt Flaschen collaborated with the Editor Engagement Features team to build notifications to welcome new editors and invite them to contribute via the Getting Started.

For the account creation and login work, S Page, Munaf Assaf, and the rest of the team rebuilt our design to work with MediaWiki core, and solicited reviews from outside the team. We currently plan to launch both interface redesigns on an opt-in basis in April, to have editors test the localization and other functional aspects of the forms via a URL parameter, before we enable them as default.

2013-04-monthly

In April, the Editor Engagement Experiments (E3) team focused first and foremost on its account creation and login redesigns in MediaWiki core. The first phase of the launch invited editors and readers on all Wikimedia projects to test the new forms on an opt-in basis, to identify bugs and localization issues across our many wikis. We expect to release these as the default forms in May, pending any final blockers.

For the team's Onboarding new Wikipedians project, we completed quantitative analysis of the latest version of the GettingStarted landing page, and began prototyping a new landing page and navigation system for usability testing prior to further development and launch, which is expected in early May as well.

On the analytics and infrastructure front, the team handed off the product roadmap for the User Metrics API to the Analytics team and colleagues in the Grantmaking and Programs department. Ori Livneh, in support of the data analysis needs on the team, began work supporting a Foundation instance of IPython Notebook.

Last but not least, the E3 team held its second Quarterly Review session, and began work planning its next high-level goals for the April–June quarter.