Editor Engagement/2013 strategy planning (Features)

This is a strategy planning space for the Editor Engagement Features team in 2013. Our initial focus is to plan the third quarter of the Wikimedia Foundation's 2012-2013 fiscal year (January-March 2013). (See also: these planning slides, which will soon be published on Commons)

Goals
Our overall goal for 2012-2013 is to help reverse Wikipedia's editor decline by recruiting new editors and better retaining existing editors.

(See also: this excerpt from WMF's 2012-2013 Plan.)

We are working on three projects this quarter:
 * Echo (Notifications)
 * Article Feedback v5
 * Flow (Messaging)

Note that we have moved all Article Feedback goals to this separate AFT planning page, since that project is further along and can be discussed with a smaller team. And this page doesn't cover Flow in any depth, as this project is still being defined as of this writing.

Here are proposed quarterly goals for January-March 2013:
 * Provide a first set of notifications to engage new editors to contribute more to Wikipedia.
 * Engage more readers to use Article Feedback to contribute and register on Wikipedia.
 * Continue design work for Flow, our new user-to-user messaging system

Milestones
And here are proposed quarterly milestones for January-March 2013:
 * Expand first beta of Echo (Notifications), with better features for new editors
 * Deploy a limited release of Echo on the English Wikipedia, focused on new editors
 * Complete Article Feedback v5 features and code refactoring, with limited testing
 * Deploy AFT v5 to 100% of the English Wikipedia, using a special database cluster
 * Refine Flow design (user-to-user messaging), solving interdependencies with Echo and talk pages
 * Start infrastructure development to support wide-scale user-to-user messaging

Target
The WMF plan identified a target of 86,000 active editors to Wikimedia projects by end of June 2013 (from 85,000 in March 2012).

This target was based on the number of editors who make 5 edits per month for all projects except Commons. As of October 2012, WMF reported 80,000 active editors for all projects, of which 8,000 were for Commons.

Note that WMF has modified its metrics tracking methodology for active editors, which means that the original 86k target would need to be adjusted down in order to match the current stats generated by our new system and quoted above. We expect these numbers to be adjusted in coming weeks.

Research
Some of these research metrics and findings can help inform our work this quarter.

Funnel
Here are some rough estimates of our monthly user funnel for the English Wikipedia:
 * Readers:	~300M
 * New accounts:	~150K
 * New editors (10+ edits):	6K
 * Active editors (5+ edits/month):	32K
 * Very active editors (100+ edits/month):	3.2K

While these are just estimates from a range of different sources, they give us a sense of scale. For example, there seems to be on the order of 9,000 readers for each active editor on the English Wikipedia. And only about one in 25 new members who register go on to become auto-confirmed editors (with 10+ edits).

These examples suggest a large untapped potential for editor engagement, in three distinct groups: readers, new members and new editors.

(Source for editor counts: Wikimedia Report Card for October 2012)

Editor Decline
We’re currently seeing a 0.5% y-o-y decline in active editors for all projects combined from Oct. '11 to Oct. '12. We're also observing a larger 5% y-o-y decline in active editors for the English Wikipedia for the same time period (34.3K in Oct. '11 vs. 32.5k Oct. '12), according to the Wikimedia Report Card for October 2012.

As of October 2012, WMF reported 80,000 active editors for all projects, of which 8,000 were for Commons, which could mean a net 72,000 active editors for all projects other than Commons. This suggests a gap between this report and the WMF's target of 86,000 active editors by July 1, 2013, but part of this gap is due to changes in metrics methodology, as explained above.

Reversing this decline will take sustained effort on multiple fronts, from editor engagement features and experiments to visual editor, internationalization and site performance improvements.

New Members
One of our best opportunities for engaging new users is by inviting them to first create an account, so we can effectively communicate with them through notifications and other tools. And we are signing up on the order of 125-150k new users per month on the English Wikipedia alone, which is significant.

Yet our research suggests that about 65% of the new users who register end up not making any edits after they sign up.

It seems that a few well-designed notifications could invite these new users to contribute and could help convert them into editors. Otherwise, notifications would have almost no impact at all on these new members, which would be unfortunate.

To address this finding, we recommend a discussion of how to make the watchlist (and some of the "notifications" inherent in watchlist) more visible to new users. At a minimum, we want to let users know that the watchlist exists and potentially make some preference changes to make watchlists more consumable. Ideas worth considering include informing new users on how to use the watchlist, and/or automatically add pages that you start or edit to your watchlist.

Examples of such invitations could include a reminder that they can edit, an invitation to check their watchlist, suggestions of articles to improve, tips on how to edit, etc.

First Edits
Last year, Maryana Pinchak did a study of first edits for 100 users, to better understand how we can support them as they start their journey as Wikipedia contributors. This led us to map out that journey on this board, which also shows what types of contributions they made after their 100th edit.

Here's a summary of their first edits:
 * Edit an article:	74%
 * Create a page:	16%
 * Create user space:	7%
 * Comment on talk page:	4%

Some of this behavioral data can help inform our work on notifications, given our focus on new editors for the Echo project. We want to make sure that the notifications we send are mapping well with these users' first steps.

For example, we now only offer edit notifications if a user's edit is reverted, and currently do not send any positive or neutral notifications for any other events that relate to that edit. And the edit reversion notification now sends these new users to the diff page, which is very confusing. Both of these issues would seem like a very high priority for our work this quarter, in order to support these new users during their critical first steps.

We recommend providing a "happy path" for new editors over time, using one or more positive notifications once they have completed an unreverted edit. The rationale for this is that oftentimes, this user receives no response from the community and it would be good to introduce some positive feedback. The assumption here is that under some definition of unreverted, these users are well-intentioned and could become constructive editors.

Some of the other details in this research can help us determine when to send follow-up notifications to these new users.

Feedback Registrations
Our latest research suggests that Article Feedback appears effective in getting new users to contribute to Wikipedia, as summarized on the WMF blog (see ).

For example, 2.7 percent of readers who post feedback go on to create a new account, after being invited to sign up. And 3 percent of these new users go on to edit articles within 24 hours from signing up.

At this rate, we project that Article Feedback can generate several hundred thousand new registrations per year on the English Wikipedia, which is significant. While we cannot accurately estimate how many of these new contributors would become active editors (5 edits/month), it appears that this program can have some impact in reversing the current editor decline, if combined effectively with other engagement programs such as notifications.

Feedback Moderation
Our research suggests that up to 40% of the reader feedback is found useful by editors, but it is buried under a lot of noise, which requires some form of moderation and/or filtering. But editor moderation activity is quite low when compared to the volume of feedback posted every day. Less than 10% of feedback posted every single day receives any kind of moderation within a month, whether by readers or registered editors.

These issues could be addressed by better moderation tools and filters, both automated and manual. And many editors can’t easily find comments for articles they edit, an issue which could be addressed by making feedback more visible to editors on the article pages.

Messaging
We don't currently have any new research to share about Flow, our user-to-user messaging initiative, since this project is still in design phase.

As stated in our Engineering Goals for 2012-13, the current talkpages used by Wikipedia editors to communicate have many serious problems. Among the issues:
 * there are unclear social conventions of how to respond
 * there is no single canonical place for this interaction to live
 * the concept of a conversation is vague (both from a user and technical perspective), notifications (see above) to all parties are crude/non-existent.
 * the UI does not represent anything resembling a modern discussion system (does not resemble any existing web affordance).
 * there is no consistent or relatable framework for surfacing a new messaging event (update) to the user (in the form of a notification)

As messaging actually covers a wealth of user-user interactions and requires a notification system to close the loop, the primary focus for this 2012-13 fiscal year is only on a single user-user "talk page"-like discussion, spec'd out in a "mobile-first" design manner, to create realistic constraints on the complexity of the problem.

Also note that this project is expected to require significant back-end infrastructure work before we can actually develop and test it.

Next steps
Our current development work on E2 has been significantly delayed, and is at least 3 months behind the schedule proposed in WMF's Engineering Goals for 2012-13. We will want to revisit this original plan and determine more realistic milestones together in coming weeks.

For now, this document is intended to help us identify objectives we want to focus on in this quarter (Jan. - March 2013). We will schedule a quarterly planning meeting with our entire team in coming days. In the meantime, we would like to invite team members to suggest ideas on this page, in advance of the planning session.

To get the ball rolling, here are a few questions about core feature areas, which we would like to discuss with you for each product.

Echo (Notifications)

 * What are the next notifications we should develop, considering their importance to new editors?


 * Can we get notifications to map more closely with the new editor's actual experience? How can we better inform you of events related to your first edits?


 * Should we consider notifications that invite newly registered users to contribute, even if they do nothing after sign up?


 * Should we consider moving all talk page notifications into a separate stack, with its own counter and/or flyout?


 * Should we make watchlist notifications more accessible? should we consider improvements to the watchlist?


 * Should we look at quick solutions that complete the user's journey? (e.g. revert notifications now send users to the cryptic diff page)


 * Should we initially deploy notifications only for new editors on English Wikipedia? or should we consider another way of limiting this experimental release?


 * How should we develop a robust queuing system and HTML email support in a timely manner, so we can deploy on English Wikipedia in this quarter?

See our feature requirements for early solutions to some of these questions.

Article Feedback

 * How can we best improve our filtering tools to automatically surface more good feedback? to remove irrelevant comments?


 * How can we make useful feedback more visible to editors? through a special link on article pages? with a 'post to talk page' feature?


 * How can we reduce the editor workload? with simpler feedback moderation tools? with better guidelines?


 * How can we make this tool more scaleable, so it can support millions of comments? with a special database cluster? through a sharding tool?


 * How can we collaborate with WMF Ops to insure a timely deployment of this tool on a special database cluster for now?


 * How can we socialize this project with Wikipedia editors, to promote its benefits and address any community concerns?

See our features under consideration for solutions to some of these questions, which we identified as a team last quarter.

Flow (Messaging)

 * How is our current design progressing for this initiative? How much development would be required for the current design?


 * How will user-to-user messaging features work?


 * How does this new feature co-exist with talk pages?


 * What are the interdependencies between Echo and Flow?


 * How can we build an infrastructure that can support user-to-user messaging given our current technical constraints?

See our project page for early solutions to some of these questions.

Active Editor Target

 * What would be reasonable targets for our E2 editor engagement features in this quarter?

For this question, let's consider the following points:
 * the editor decline has increased on some projects since the current active editor target was originally established
 * this target is shared by many other groups besides the E2 team: what can we contribute towards this company-wide target?
 * we have experienced serious delays that make some of our original goals impractical

Feedback
Please use the discussion page to leave comments, suggestions, or questions about this project.