Growth/Growth team updates

This page contains updates on the Growth team's work to increase new editor retention in Czech and Korean Wikipedias. The planning for this work began in June 2018. The team works on multiple projects at a time, but they will all be discussed on this page. During July, August, September, and October 2018, the team was also simultaneously working on a project to improve the Articles for Creation process and New Pages Feed in English Wikipedia, which is discussed on its own page.

How to get involved
It is important that our work is grounded in the reality of the communities we hope to help. If you have thoughts or ideas around this project or our team's work, please comment on this project's talk page.

Update 2018-11-25: two projects deployed
We're excited to have deployed two of this quarter's three projects! The EditorJourney schema is now collecting data on the activities of new users in Czech and Korean Wikipedias, and the Welcome Survey is being shown to new users in both those wikis (but only to half of those new users, because half are in the control group). These deployments are our team's biggest milestones to date, and we're excited to start learning from the incoming data. As that data comes in, we'll be turning our engineering attention to the Help Pane project, and to the next version of the Welcome Survey.

Personalized first day (Welcome survey)


 * We deployed this on November 20. This was two weeks later than our initial plan of November 6.  The main reason for the delay was the translation of the survey privacy statement and the privacy policy into Czech and Korean.  The external translation vendor took one week to translate the documents, and then our ambassadors needed an additional week to adjust and correct parts of the translations to use wiki language.  For future projects, we will be mindful of finding out in advance all the legal documents that are needed, and planning for the two weeks it takes to finalize strong translations.  We also now have a set of improved best practices for working with translation vendors.
 * The survey is now being shown to half of new users in both Czech and Korean Wikipedias, and responses are being recorded in the database (the other half of new users are the control group in our A/B test, where we are checking to make sure the survey does not depress the rate at which newcomers make edits). In the next week, we'll take an initial look at the data to identify any issues or trends.
 * There are two next steps for this project:
 * Variation C: during our design research for this project, we did a comparative review of how other platforms find out basic information about their users. From that, we created an ideal design of how the Welcome Survey would look, which we called "Variation C".  We think that design will maximize the number of users who complete the survey and stay on the wiki.  But instead of building that design first, we quickly deployed the more simple "Variation A", so that we could start gathering data.  Now that Variation A is deployed, we have turned attention to upgrading to Variation C over the course of the next two to three weeks.  We plan to A/B test the two variations to see if they cause different response rates or have different effects on whether new users stay on the wiki.
 * Personalizing: the original objective of this project was to gather some initial information on what new editors are trying to accomplish so that we can give them the help content, task recommendations, or mentorship they need to be successful. We're currently discussing the right first project to work on to take action on the data we're collecting.  Some of the options we're thinking about are on the short list below.  But each of those things would need a place to "live", and this is making us think about ideas for some kind of "newcomer portal" where newcomers would easily be able to find the tools and information they need.  We'll develop this idea a little more, and propose some specifics so that experienced community members can react and tell us what they think.
 * Making it easy for newcomers to see editing activity around the topic areas in which they indicated that they're interested.
 * Connecting interested newcomers to experienced editors.
 * Surfacing the help content most relevant to the reason for which the newcomers created their accounts.

Understanding first day (EditorJourney)


 * We deployed this on November 15. This was nine days later than our initial plan of November 6.  The main reasons for the delay were some additional security reviews done by engineers from outside our team, and on the rigorous testing and adjusting we did before deployment.  We're glad that the additional security reviews took place, because it's critical that our software is safe, but we'll know for the future to incorporate our dependencies on other teams into our planning.  The rigorous testing and adjusting took longer than expected because of the many idiosyncrasies and edge cases in the Mediawiki software: we kept finding ways that the event data wasn't being recorded correctly.  We think we have now solved the vast majority of those cases.
 * Data is now being recorded for all new users in Czech and Korean Wikipedias, and we've been auditing the data and preparing to make initial reports.

Focus on help desk (Editor help pane)


 * Engineers are now working on building the initial iteration of the help pane, though most of our attention has been focused on completing the roll out of the two other projects listed above.
 * We are currently planning to deploy this during the week of January 7. We think our engineering work will take up until the holiday season, and we don't want to deploy something new when many engineers and support staff will be taking vacations.
 * In the coming week, we'll be drafting the written copy that will be part of the feature. This copy is important, because we want to make sure new users understand what will happen when they ask a question in the help pane.  That work is happening in this task.
 * We're also going to be running some live user tests, like we did for the Welcome Survey.
 * In addition to giving the ability to ask a question, the help pane will also contain a set of links to existing help content. Our ambassadors are determining the right initial set of most helpful links in this task.
 * We have some open questions around the rules and design choices for this feature. Later this week, we'll post them and start a community discussion to get recommendations from editors who have experience working with newcomers.

Update 2018-11-13: final testing on two projects, work begins for "Editor help pane"
A quick note before the update is that we've given nicknames to our projects, since "Personalized first day", "Understanding first day", and "Focus on help desk" are difficult to say and type. While those phrases still represent the overall goals of the projects, we're now also referring to them on a day-to-day basis as "Welcome survey", "EditorJourney", and "Editor help pane" (respectively), which are more specific to what we'll be deploying in the near term around those three projects.

Personalized first day (Welcome survey)


 * Current planned release to Korean and Czech Wikipedias: Nov 15
 * The welcome survey is now deployed to our testing environment where we are making sure that all the questions are laid out as expected, that no words are being cut off, that data is being recorded correctly, and that all the business rules we planned out are being followed. We'll need to test and fix for a couple more days.
 * The translations of the survey privacy statement and full privacy policy have been returned from our translation vendor, and are currently being edited by our community ambassadors from Czech and Korean Wikipedias to make sure that the vocabulary used makes sense. Posting these is also a blocker for release.

Understanding first day (EditorJourney)


 * Current planned release to Korean and Czech Wikipedias: Nov 15
 * Several team members are using Test Wiki and Beta Cluster to make sure that this data is being recorded correctly for both the desktop and mobile experiences and that all the privacy safeguards we put in place are working right. For instance, we're making sure that when users read articles in the main article namespace, the actual title of the article is not saved.  Testing EventLogging is generally tedious, because it involves clicking through as many different elements of the software as possible, and then checking the database to make sure the actions were recorded correctly.  A few bugs and issues have been found so far and are in the process of being fixed.  The specifics can be found on this Phabricator task.

Focus on help desk (Editor help pane)


 * The team completed an initial design process for this feature in which we dreamed big, and looked into the future of what this feature could be if it shows a lot of promise. To do this, our designer drew on the review of similar features in other software and on similar Mediawiki projects from the past.
 * After going through all the ideas that came out of the design process, we decided on what the initial version of this feature would contain. This Phabricator task lists all the subtasks that describe the various design ideas.  These two are the ones we'll be doing for the initial version (T206717 and T209318), in the points below.
 * A button in the lower right for newcomers, giving them the opportunity to "Get help with editing".
 * Clicking the button opens up the "help pane".
 * The help pane contains a set of links to the most useful help pages in the wiki.
 * The help pane also contains an option to "Ask a question", in which newcomers will be able to type a question that will be posted automatically in their wiki's help desk.
 * We're starting to engineer on those initial requirements and hammering out many small details. Please check out the evolving project page to follow along and see mockups of what this will look like.

Update 2018-11-02: nearing deployment on two projects
For both the "Understanding first day" and "Personalized first day" projects, we are nearing deployment within the next two weeks. Both projects are getting their final approvals and are being tested. Next week, we'll post a detailed update on Czech and Korean Wikipedias explaining what will be changing in the wikis and when.

Since those two projects are on the verge of being deployed and beginning to collect data, our team is going to be turning attention to the "Focus on help desk" project. We'll be making a full project page for that within a couple days so that it is easier for the community to follow along with how we're thinking about it.

Personalized first day


 * We settled on paths forward for the open questions from last week's update:
 * Topics of interest: since we noticed that WikiProjects differ substantially from wiki to wiki, and since we are not completely sure how we plan to use the new user's topics of interest, we assembled a list of 27 topics that appear across many different lists of important topics, including Czech WikiProjects, Korean WikiProjects, English WikiProjects, categories, and Vital Articles. The hope is that many new users will find find a topic that interests them, and if not, they can type in their own topics.  We will analyze the results to decide what to do next.  The topics we chose are Arts, Science, Geography, History, Music, Sports, Literature, Religion, Popular culture, Entertainment, Food and drink, Biography, Military, Economics, Technology, Film, Philosophy, Business, Politics, Government, Engineering, Crafts and hobbies, Games, Health, Social science, Transportation, and Education.
 * Interest in mentorship contact: we decided on phrasing the question such that it's clear that we do not yet have an active mentorship program. This question is now phrased as, "We are considering starting a program for more experienced editors to help newer users with editing. Are you interested in being contacted to get help with editing?"
 * We incorporated results from human testing that occurred through usertesting.com. Details on this will be added to the project page.
 * The finalized survey copy is now in translatewiki.net, and is already translated into Czech and Korean (as well as many other languages).
 * We have finalized a survey privacy statement for the survey, and we are in the process of getting it and our full privacy policy translated into Czech and Korean. This will help new users understand how their responses to the survey will be used.
 * We have decided on some of the details for the A/B test we'll be running with the feature. One of the community's main concerns with this feature is the potential that a longer sign-up process would cause new uses to leave before editing.  Our experiment will give the survey to some new users, and leave the experience of others unchanged.  Essentially, if they continue to edit at the same rates, than we can conclude that the survey is not causing users to leave before editing.
 * Up next, we'll be deploying the survey to our testing environments and checking to make sure everything works as expected.

Understanding first day


 * We've completed privacy review, and we are confident that we'll be handling user data in a way that helps our team learn while respecting user privacy.
 * We are finalizing some of the engineering and security aspects of this work, after getting review from several engineers around WMF.
 * We are now testing that data is being recorded and obfuscated correctly by using our test environments.

Focus on help desk


 * This project is in the midst of being designed right now. To aid in the design, the team has been:
 * Reviewing the way that other platforms and software offer help
 * Reviewing similar Wikimedia projects from the past (such as MoodBar and the Article Feedback Tool) to make sure we incorporate learnings from those efforts.
 * We expect to be able to start engineering on this project next week.

Update 2018-10-25: new project pages and open questions
Over the last week, we have made a lot of progress on the "Understanding first day" and "Personalized first day" projects. We currently hope that we can deploy both of those projects to some extent during the week of November 6th, but there are still many details to resolve in the next two weeks. We are hoping for any community wisdom to help us resolve some of the open questions laid out below. To help make it easier to follow along with our work, we now have full project pages for each of those two projects:


 * Personalized first day
 * Understanding first day

Those pages contain things like designs, decisions, and implementation details. The project page for "Focus on help desk" is coming next.

Below are some updates on the two projects above.

Personalized first day

Our team has been iterating on the exact wording for the text of the questions and responses in the survey. The mockups now reflect more up-to-date question wording, but we're still finalizing it. We've asked some survey experts to weigh in, and we have also engaged some human testers to try out the surveys and give feedback. We have a couple of open questions that we're not sure how to resolve yet:


 * Topics of interest: we want to ask new users about the topics they're interested in editing. That's because one day we may want to recommend editing tasks that fit their interests, direct them to WikiProjects about their interests, or connect them with other editors who share their interests.  We are trying to figure out what topics to offer as choices.  One idea is to use WikiProjects, but we know that the majority of WikiProjects tend to be inactive.  We are also concerned that presenting very general topics -- "Art", "Sports", or "Geography" --are not specific enough to align with someone's actual interests, which may be "Ceramics", "FC Barcelona", or "Sahara desert".  We're still gathering ideas and experimenting with the right way to do this.
 * Interest in mentorship contact: one of the questions on the survey asks whether the user is interested in being contacted by an experienced editor for help with their editing. This is important because it will help us gauge whether many new users are interested in mentorship, which might encourage our team to develop ideas around promoting mentorship on-wiki.  We are trying to figure out a good way to ask that question without making new users wait to receive contact that won't be coming immediately, since we haven't yet developed the way to actually connect them with experienced editors.  We don't want our questions to be confusing or misleading.

We are also in the process of making sure that we have thought through any data privacy elements with the survey so that users can feel good about answering the questions.

Understanding first day

We have largely settled on exactly what data we will be collecting and the questions we plan to answer with it. The project page describes those questions, and the exact technical specifications for measurement can be found here on Meta Wiki, with the associated business rules on its talk page. In short, we'll be recording the pages that new users in Czech and Korean Wikipedias visit during the first 24 hours after creating their accounts -- but only pages in non-sensitive namespaces that are important for our work, such as the Help, Wikipedia, Special, and User namespaces. For pages in the Article, File, Portal, and Draft namespaces, however, we will only record that a visit to that namespace occurred -- not the specific page. The identifying data will be deleted or aggregated after 90 days.

This new data will allow us to see the path new users take as they learn and prepare to edit. We're also going to connect the new data to existing measurement efforts in the visual and wikitext editors so we can see how the new user's path leads them to succeed or not succeed with their first edits.

Most of the technical work is complete, but we are still in the process of security, engineering, and privacy checks on our work. We're testing out the instrumentation in testing environments, and making sure that the other measurement efforts we want to use have the data we'll need.

Update 2018-10-18: starting three projects
Our team has set goals for what we'll accomplish by the end of the quarter (we are currently in 2018 Q2, which goes from October 2018 to December 2018.) Our goals can be found with the rest of the Audiences department's goals here. In brief, our goals for this quarter are:


 * Collect initial "Understanding first day" data from instrumentation, and post early numbers on wiki.
 * Collect initial "Personalized first day" data from user input, and post early numbers on wiki.
 * Deploy a "Focus on help desk" intervention experiment in production.
 * Develop a relationship with a third community beyond Czech Wikipedia and Korean Wikipedia.

Since choosing those three priority projects at the beginning of October, the Growth team has started work on each one, with most of our activity taking place in Phabricator under an "epic" task for each project:


 * Understanding first day
 * Personalized first day (this project is the first to also have a project page)
 * Focus on help desk

Over the last two weeks, the team has primarily been working on "Understanding first day" and "Personalized first day".

For "Understanding first day", our engineers and data analyst have been developing the specifications for an EventLogging schema to record useful information on new editor journeys while respecting their data privacy. Our team is close to posting the specifications on wiki and in Phabricator, and it will be summarized in an update here.

For "Personalized first day", we have designed initial concepts for how this set of questions will be asked to the new account holders, and the engineers are beginning to prototype them. It's possible to explore those mockups here -- but it is important to note that the specific language in the mockups has not yet been worked on. The language in those mockups is placeholder language for when we write the finalized words that will be translated into Korean and Czech. Writing those words is our current highest priority task for this project.

Background
The Growth team is very appreciative of the time that community members took to respond to our ideas for increasing new editor retention. We received helpful comments and opinions from around the world, both from newcomers and experienced editors, that have allowed us to put together our plan for the next several months of work. Here is how we have proceeded (along with relevant links):


 * 1) We developed an understanding of new editor experiences and challenges through the New Editor Experiences project.
 * 2) We put together a list of ideas for how to increase new editor retention.
 * 3) These ideas were discussed in many different communities in three different languages: English, Czech, and Korean.
 * 4) The Growth team met to discuss and summarize all the community feedback.
 * 5) The Growth team made the following development plan that takes into account community thoughts and several other factors.

Theme
The theme of our work for the coming months is “Day One”. That means we are focusing on the experience of new editors on the day that they create their account. We’re focusing on day one because our data shows that about two thirds of people who create accounts never make an edit, and of those who do, about two thirds of them never come back to edit again. This makes us think that the first day of a new editor’s experience is an important time to help them start editing and want to continue editing. We will go into more detail on the data behind this in another update.

Plan
With the theme of “Day One” in mind, we decided to prioritize the following three ideas for Czech and Korean Wikipedias. We are not building any of them fully. Instead, we will build a small piece of each so that we can learn whether the idea has potential, and quickly decide whether to continue. We’ll be working on them in October, November, and December 2018.


 * Understanding first day: we are prioritizing this because it will help us learn a lot about new editors.  It will not change any user’s experience. We’ll simply add logging so that we can see what new editors do right after creating their accounts.  We will be careful with user privacy, and we hope to share initial results in December.
 * Personalized first day: this idea will also help us learn a lot about new editors.  To start, we will add some optional questions to the new editor’s registration process.  This will help us learn what new editors are trying to accomplish, and by looking at the data, and with the help of the communities, we’ll be able to think about how to personalize a new editor’s experience so that they can accomplish what they’re trying to do.  We hope to share initial results in December.
 * Focus on help desk: we are prioritizing this because it was shown to work in other contexts.  We will likely direct traffic to existing Help Desks through a bot that invites new editors on their talk pages, or through a new button in the editing experience inviting new editors to “Ask a question”. That button would be an opportunity to learn whether the “In-context questions or chat” idea has potential.  We hope to have an initial experiment running in December.

We decided to prioritize those ideas for these reasons:


 * They have support from multiple communities.
 * They address issues brought up in the New Editor Experiences research.
 * They have been proven to work in Wikipedia and non-Wikipedia contexts.
 * They help us learn quickly.
 * They can be deployed in small pieces without a major commitment at the beginning.
 * They can be deployed relatively easily to other communities.

Next steps
Next, we will be planning the work for each of those three ideas and beginning to write code. We’ll continue to post updates, and we will be clear when we need help from community members. We have already started to create Phabricator tasks for the "Understanding first day" work. For the “Focus on help desk” project, we will need experienced editors to dedicate time to answering questions in the Help Desk, because we will expect many more new editors to be asking questions. We’re looking forward to continuing to work with the Czech and Korean communities, and we are looking for additional communities to begin to work with.

Update 2018-09-21: summaries of community discussion and planning sessions
Over the past few weeks, we received a tremendous amount of useful thoughts and feedback on our list of eight ideas. We're grateful to the members of the many wiki communities (including several mid-size wikis) who responded on the talk page, on their own wikis, or sent us their thoughts through other channels. We were also able to get feedback on the ideas from actual new editors in Czech and Korean Wikipedias, who provided a first-hand perspective on which ideas would have helped them as they were starting out. Summaries of the feedback we received on each idea are now posted, along with some follow-up questions on the talk page.

To take action on this feedback and plan the coming months, the Growth team came together in person last week. We did two main things: short-term planning and long-term thinking.

Short-term planning To determine what our team will work on first, we spent a day processing the community feedback and reflecting on the eight ideas. We split up into groups, each of which was assigned two of the ideas. For each of their two ideas, those groups read through and discussed community comments in detail. They then summarized the comments into a poster to explain the nuanced perspectives to the rest of the team. This format helped us discuss each idea, making sure to take into account the pro's and con's from different communities. Posted here is an example of one of the posters we made, to give a sense of our process. To read summaries of what we learned from communities, see this page.

Long-term thinking

The Growth team intends to be working on new editor retention for many quarters, and we know that we will need to be continuously learning about new editors in order to be successful. Though we already know a lot about new editors from research done in the past year, there are still many open questions about how new editors work and what they need to be successful. We spent time determining which questions are most important so that our team can use upcoming opportunities to find answers. Please visit this page to see our most important open questions and read about the exercise we did to generate them.

Next steps

We are now in the process of deciding exactly which things to build and in what order, taking into consideration engineering challenges, measurement challenges, and how to work with the Czech and Korean communities. Our next update will contain our plan for the coming months.

Update 2018-09-04: final week of community discussion
Over the past two weeks, we have posted the list of ideas for discussion in many places, and we would like to gather all the feedback we can by the end of this week from as many different communities as possible. If you have planned on weighing in, please do so in the next few days!

Next week, the Growth team will be taking a substantial amount of time to process and digest all the thoughts and reactions that have been posted, and we will make a plan for which ideas make the most sense to pursue, and we'll post that here. In making that plan, we'll take into account community thoughts, technical considerations, and the burden that new development will put on existing communities.

Here are the places we have posted the list of ideas for comment:


 * In English on mediawiki.org.
 * In Korean on Korean Wikipedia.
 * In Czech on Czech Wikipedia.
 * It has been linked on Serbian Wikipedia.
 * It has been linked on Slovak Wikipedia.
 * It has been linked on many Help Desk talk pages in different wikis.
 * It has been sent to community members, newsletters, and Facebook groups in the Czech and Korean communities.

Update 2018-08-22: community discussion begins on ideas
We've now posted a list of the main ideas we're considering for the team to work on first! We hope as many people from as many communities as possible participate in the discussion on that talk page. The list of ideas is based on a lot of research and conversations the team has been taking in, but they are by no means solidified -- that's why this community discussion is really important to us. We're hoping to hear the majority of the community thoughts within the next two weeks (by 2018-09-05).

The list is also being translated in Czech and Korean and discussed on those wikis, since those are the places where the features would be deployed first. If you are interested in translating all or part of the list for your own wiki, please let us know! It would be great to have additional languages discussing.

We'll also be sending out our first team newsletter about this. Please sign up here if you are interested in receiving the newsletter.

Update 2018-08-14: narrowing down on ideas
The team is now spending an increasing amount of time planning for this first project, even as our engineers work on the Articles for Creation and New Pages Feed project for English Wikipedia. A few short updates are below, with more extensive information coming over the next week:


 * Our top priority right now is to start the conversations around which feature ideas we should pursue first. The team has been compiling many ideas for what we can build and has started to narrow to the ones that seem to have potential.  We're going to be asking the Czech and Korean communities to weigh in, and we will also post them here and in other venues so that as many communities as possible can add their thoughts.  We'll be looking for new ideas, thoughts on existing ideas, and information on when similar things have already been attempted.  Below are some examples of the types of ideas we're talking about.  More detail will be posted here in the coming weeks, as well as the reasons we think these ideas could be impactful.
 * Inviting new editors to help desks. Most wikis have a help desk, but most newbies don't know about it.
 * Making it possible for newbies to ask questions in the context where they are editing, instead of having to go to a different page.
 * Communicating a new editor's impact to them via email, perhaps by telling them about the page views on pages they've edited.
 * Gathering optional information from new editors when they create accounts about what they are trying to do and the topics in which they are interested, so we can direct them to information that serves their needs right away.
 * Instrumenting the new editor experience to learn more about what new editors do right after creating their accounts -- whether they read documentation, attempt to edit but don't succeed, or something else.
 * We have completed the staffing of our team by adding Rita Ho, our user experience designer, and Morten Warncke-Wang, our product analyst. Rita joins us from the team that built the Wikipedia Android app, and has important expertise on mobile usage that will be relevant for new editors.  Morten is a longtime researcher of Wikimedia projects and has deep experience understanding the issues newbies face and experience working with Mediawiki data.  Morten is going to be working on the "analyzing data" work mentioned in the update from 2018-07-06 below.
 * Several of our team members attended Wikimania 2018 and had great discussions with editors from many mid-size Wikipedias, including Arabic, Bengali, Serbian, Ukrainian, and others. We gathered information on the experience of new editors in those wikis, what is the same and what is different across them, and what specific tactics and features are working well to retain new editors.  We'll be assembling and posting what we learned, along with the work our Czech and Korean ambassadors have already done on that front to catalog existing help content in their wikis.

Update 2018-07-06: data, planning, and research
While the majority of the team's effort is currently going toward the Articles for Creation and New Pages Feed project for English Wikipedia, the team is laying the ground work for this initial Growth project by doing three main things:


 * Analyzing data about contributor behavior in Czech and Korean Wikipedias. We are setting up the queries and reports that will allow us to track the contributor retention rates in our target wikis, and to understand exactly where in the editing process new contributors are most likely to drop off.  We will post this data on wiki as soon as we're able.
 * Partnering with our ambassadors in the Czech and Korean communities to document the current resources that help new contributors in those wikis. We want to make sure that anything we build compliments the existing pages and tutorials that already exist, and that we are not duplicating efforts with any community members.  This is the related Phabricator task.
 * Summarizing existing ideas for features that could increase retention. Many WMF staff and community members have attempted and proposed ideas for new editor retention in the past.  We're making sure to collect those ideas and use them as a starting point.  We will post that information on wiki as soon as we're able.

We have also spent time this week establishing the processes for our new team, such as when we have meetings, how our Phabricator board works, and how we'll balance new work with maintenance on previous features built by engineers on the team. Those previous features include Notifications and the new filters on the Recent Changes and Watchlist pages.