Editor Engagement/2013 strategy planning (Features)

This is a strategy planning space for the Editor Engagement Features team in 2013. Our initial focus is to plan the third quarter of the Wikimedia Foundation's 2012-2013 fiscal year (January-March 2013).

Goals
Our overall goal for 2012-2013 is to help reverse Wikipedia's editor decline by recruiting new editors and better retaining existing editors.

(See also: this excerpt from WMF's 2012-2013 Plan.)

Here are proposed quarterly goals for January-March 2013:
 * Provide a first set of notifications to engage new editors to contribute more
 * Engage more readers to use Article Feedback v5 to contribute and register
 * Start developing the infrastructure for a new messaging system

Milestones
And here are proposed quarterly milestones for January-March 2013:
 * Expand first beta of Echo Notifications, with better features for new editors
 * Deploy a limited release of Echo on the English Wikipedia, focused on new editors
 * Complete Article Feedback v5 features and code refactoring, with limited testing
 * Deploy AFT v5 to 100% of the English Wikipedia, using a special database cluster
 * Refine Flow design (user-to-user messaging), solving interdependencies with Echo and talkpages
 * Start infrastructure development to support wide-scale user-to-user messaging

Target
The WMF plan identified a target of 86,000 active editors to Wikimedia projects by end of June 2013 (from 85,000 in March 2012).

This target was based on the number of editors who make 5 edits per month for all projects except Commons. As of October 2012, WMF reported 80,000 active editors for all projects, of which 8,000 were for Commons.

Research
Some of these research metrics and findings can help inform our work this quarter.

Funnel
Here are some rough estimates of our monthly user funnel for the English Wikipedia:
 * Readers:	~300M
 * New accounts:	~150K
 * New editors (10+ edits):	6K
 * Active editors (5+ edits/month):	32K
 * Very active editors (100+ edits/month):	3.2K

While these are just estimates from a range of different sources, they give us a sense of scale. For example, there seems to be on the order of 9,000 readers for each active editor on the English Wikipedia. And only about one in 25 new members who register go on to become auto-confirmed editors (with 10+ edits).

These examples suggest a large untapped potential for editor engagement, in three distinct groups: readers, new members and new editors.

(Source for editor counts: Wikimedia Report Card for October 2012)

Editor Decline
We’re currently seeing a 5% y-o-y decline in active editors for the English Wikipedia (34.3K in Oct. '11 vs. 32.5k Oct. '12). We're also observing a smaller 0.5% y-o-y decline in active editors for all projects combined for the same time period, according to the Wikimedia Report Card for October 2012.

As of October 2012, WMF reported 80,000 active editors for all projects, of which 8,000 were for Commons, which could mean a net 72,000 active editors for all projects other than Commons. This suggests a wider gap than anticipated between this report and the WMF's target of 86,000 active editors by July 1, 2013.

Reversing this decline will take sustained effort on multiple fronts, from editor engagement features and experiments to visual editor, internationalization and site performance improvements. What will be our focus for E2?

New Members
One of our best opportunities for engaging new users is by inviting them to first create an account, so we can effectively communicate with them through notifications and other tools. And we are signing up on the order of 125-150k new users per month on the English Wikipedia alone, which is significant.

Yet our research suggests that about 60-65% of the new users who register end up not making any edits after they sign up.

It seems that a few well-designed notifications could invite these new users to contribute and could help convert them into editors. Otherwise, notifications would have almost no impact at all on these new members, which would be unfortunate.

Examples of such invitations could include a reminder that they can edit, suggestions of articles to improve, tips on how to edit, how to leave talk page messages, how to use the watchlist, how to send wikilove, etc.

First Edits
Last October, Maryana did a study of first edits for 100 users, to better understand how we can support them as they start their journey as Wikipedia contributors. This led us to map out that journey on this board, which also shows what types of contributions they made after their 100th edit.

Here's a summary of their first edits:
 * Edit an article:	74%
 * Create a page:	16%
 * Create user space:	7%
 * Comment on talk page:	4%

Some of this behavioral data can help inform our work on notifications, given our focus on new editors for the Echo project. We want to make sure that the notifications we send are mapping well with these users' first steps.

For example, we only offer edit notifications if a user's edit is reverted, and currently do not send any notifications for any other events that relate to that edit. And the edit reversion notification now sends these new users to the diff page, which is very confusing. Both of these issues would seem like a very high priority for our work this quarter, in order to support these new users during their critical first steps.

Some of the other details in this research can help us determine when to send follow-up notifications to these new users.

Feedback Registrations
Our latest research suggests that Article Feedback appears effective in getting new users to contribute to Wikipedia, as summarized on the WMF blog (see ).

For example, 2.7 percent of readers who post feedback go on to create a new account, after being invited to sign up. And 3 percent of these new users go on to edit articles within 24 hours from signing up.

At this rate, we project that Article Feedback can generate several hundred thousand new registrations per year on the English Wikipedia, which is significant. While we cannot accurately estimate how many of these new contributors would become active editors (5 edits/month), it appears that this program can have some impact in reversing the current editor decline, if combined effectively with other engagement programs such as notifications.

Feedback Moderation
Our research suggests that up to 40% of the reader feedback is found useful by editors, but it is buried under a lot of noise, which requires some form of moderation and/or filtering. But editor moderation activity is quite low when compared to the volume of feedback posted every day. Less than 10% of feedback posted every single day receives any kind of moderation within a month, whether by readers or registered editors.

These issues could be addressed by better moderation tools and filters, both automated and manual. And many editors can’t easily find comments for articles they edit, an issue which could be addressed by making feedback more visible to editors on the article pages.

Messaging
We don't currently have any new research to share about Flow, our user-to-user messaging initiative, since this project is still in design phase.

As stated in our Engineering Goals for 2012-13, the current talkpages used by Wikipedia editors to communicate have many serious problems. Among the issues:
 * there are unclear social conventions of how to respond
 * there is no single canonical place for this interaction to live
 * the concept of a conversation is vague (both from a user and technical perspective), notifications (see above) to all parties are crude/non-existent.
 * the UI does not represent anything resembling a modern discussion system (does not resemble any existing web affordance).
 * there is no consistent or relatable framework for surfacing a new messaging event (update) to the user (in the form of a notification)

As messaging actually covers a wealth of user-user interactions and requires a notification system to close the loop, the primary focus for this 2012-13 fiscal year is only on a single user-user "talk page"-like discussion, spec'd out in a "mobile-first" design manner, to create realistic constraints on the complexity of the problem.

Also note that this project is expected to require significant back-end infrastructure work before we can actually develop and test it.

Next steps
Our current development work on E2 has been significantly delayed, and is at least 3 months behind the schedule proposed in WMF's Engineering Goals for 2012-13. We will want to revisit this original plan and determine more realistic milestones together in coming weeks.

For now, this document is intended to help us identify objectives we want to focus on in this quarter (Jan. - March 2013). We will schedule a quarterly planning meeting with our entire team in coming days. In the meantime, we would like to invite team members to suggest ideas on this page, in advance of the planning session.

To get the ball rolling, here are a few questions about core feature areas, which we would like to discuss with you for each product.

Echo (Notifications)

 * What are the next notifications we should develop, considering their importance to new editors?


 * Can we get notifications to map more closely with the new editor's actual experience? How can we better inform you of events related to your first edits?


 * Should we consider notifications that invite newly registered users to contribute, even if they do nothing after sign up?


 * Should we consider moving all talk page notifications into a separate stack, with its own counter and/or flyout?


 * Should we make watchlist notifications more accessible? should we consider improvements to the watchlist?


 * Should we look at quick solutions that complete the user's journey? (e.g. revert notifications now send users to the cryptic diff page)


 * Should we initially deploy notifications only for new editors on English Wikipedia? or should we consider another way of limiting this experimental release?


 * How should we develop a robust queuing system and HTML email support in a timely manner, so we can deploy on English Wikipedia in this quarter?

Article Feedback

 * How can we best improve our filtering tools to automatically surface more good feedback? to remove irrelevant comments?


 * How can we make useful feedback more visible to editors? through a special link on article pages? with a 'post to talk page' feature?


 * How can we reduce the editor workload? with simpler feedback moderation tools? with better guidelines?


 * How can we make this tool more scaleable, so it can support millions of comments? with a special database cluster? through a sharding tool?


 * How can we collaborate with WMF Ops to insure a timely deployment of this tool on a special database cluster for now?


 * How can we socialize this project with Wikipedia editors, to promote its benefits and address any community concerns?

See our features under consideration for solutions to some of these questions, which we identified as a team last quarter.

Flow (Messaging)

 * How is our current design progressing for this initiative? How much development would be required for the current design?


 * How will user-to-user messaging features work?


 * How does this new feature co-exist with talk pages?


 * What are the interdependencies between Echo and Flow?


 * How can we build an infrastructure that can support user-to-user messaging given our current technical constraints?

Active Editor Target

 * What would be reasonable targets for our E2 editor engagement features in this quarter?

For this question, let's consider the following points:
 * the editor decline has increased on some projects since the current active editor target was originally established
 * this target is shared by many other groups besides the E2 team: what can we contribute towards this company-wide target?
 * we have experienced serious delays that make some of our original goals impractical

Feature Ideas
Please use this space to enter any feature ideas for meeting our goals. They may be along the lines of any of the above research or questions, or something entirely different.