Growth/Personalized first day

This page describes the Growth team's work on the "Personalized first day" project, and contains major assets, designs, and decisions. Most incremental updates on progress will be posted on the general Growth team updates page, with some large or detailed updates posted here.

As of November 20, 2018, the Welcome Survey is deployed to treatment groups in Czech and Korean Wikipedias, and data has been collected and analyzed in this initial report in December 2018. The design that is deployed is called "Variation A". The next iteration of the design, "Variation C", will be tested against Variation A in January 2019.

Current status

 * 2018-11-20: deployed the welcome survey to Czech and Korean Wikipedias, with half of users getting Variation A and half in a control group.
 * 2018-12-22: published initial report on survey results.
 * 2018-01-16: after determining that receiving Variation A was not depressing activation rate, changed Czech and Korean such that half of users get Variation A and half get Variation C.
 * 2018-01-24: deployed the welcome survey to Vietnamese Wikipedias, with half of users getting Variation A and half getting Variation C.
 * Next: evaluating the Variation A vs. Variation C experiment to determine which has the higher response rate. Then we will choose a winner, and deploy that version to all newcomers in all our target wikis.

Summary
When new editors create accounts, they are asked for minimal sign-up information and then are redirected back to the context from which they initiated account creation. This is a missed opportunity to learn what the new editor is attempting to accomplish and to provide them direction or content that helps them achieve their goals. Therefore, the "Personalized first day" project contains two parts:


 * 1) Ask new editors additional questions during their account creation process, such as their reason for creating an account, what they are trying accomplish, topics that they are interested in, or whether they want to meet a mentor.  This part of the project is called the "Welcome Survey".
 * 2) After the login is created, we can direct the new editors to help content that is relevant for their goals, or to WikiProjects that match their interests, or potentially match them with a mentor who shares their interests.

The Growth team is building Part 1 (Welcome Survey) first during 2018 Q2 (October 2018 - December 2018). After analyzing the results, we will decide whether and how to pursue Part 2.

Why this project is prioritized
Research has shown us that new editors have their own specific objectives when they begin to edit, and if they are not able to pursue those objectives, they are not retained. We also know that new editors have to learn many technical, conceptual, and cultural skills in order to be successful editors of the wikis, and that it is best to teach these skills through "in-context" and "human-to-human" help. The "Personalized first day" project is meant to discover a new editor's goals by asking them, which gives us the ability to give them the "in-context" or "human-to-human" help that they need most to be successful. We prioritized this project because:


 * It helps us address the issues brought up in the New Editors Experiences research.
 * Community members were positive in their feedback on the idea.
 * It helps us increase our learning at this early stage of our team's work with new editors.
 * This sort of feature is almost universally common among other successful software platforms.
 * The feature can be built and adjusted incrementally, without a major investment before it starts to show results.
 * It will be easy to translate and apply to other wikis in the future.

Comparative review for Welcome Survey
To learn how best to design the welcome survey, our team's designer reviewed the way that other platforms (e.g. Reddit, Coursera, Medium) ask initial questions of newcomers. While the experience we want to give newcomers is definitely different than other platforms (we want to give newcomers an optional, lightweight, non-invasive experience), we also recognize that there are best practices we can learn from other software. The comparisons are shown in this slide deck, and the main takeaways are:


 * Modals were generally more in use or otherwise a full page overlay for forms.
 * No more than 4 questions were asked during onboarding.
 * Questions were either progressively disclosed on a single screen or otherwise shown 1 question per step with clear visual indication of the number of steps (eg. via a stepper UI).
 * Multi-question forms tended to be skip-able entirely, but with mandatory questions if the user wanted to participate.
 * An optional guided tour or further help was often provided afterwards once user was in the app.
 * Friendly voice and tone - welcoming, informal language framed as asking for information to help orient the user ("Tell us about yourself", "Get recommendations", etc.)

Design for Welcome Survey
Our evolving designs can always be found in these clickable mockups, and with additional contextual information in this Phabricator task. Our goal for the design is for newcomers to answer all the questions they want to, and to not depress the number of users who complete their signup process and get back to what they were doing.

We considered three main design approaches for this project. The new account holder lands at each approach immediately after clicking "Create your account" at Special:CreateAccount. And after completing each one, they are returned to the page from which they originally clicked to create their account. Please note that the question wording and text in these mockups does not reflect current drafts -- the mockups are more for the visuals and workflows. Question wording and text will be posted separately.
 * Variation A: a Special page containing the survey. This was the simplest to implement, and we deployed this one first on November 20, 2018.  Though this doesn't follow best practices that we learned in comparative review, it is the fastest way to get a survey up and to start learning.  It will also work automatically on mobile web and for no-Javascript users.
 * Variation B: this option shows users one question at a time, and also removes the surrounding links from the left and top navigation of the wiki for a more streamlined experience. This is inspired by the interface for the Content Translation tool.  We decided not to pursue this variation at all, since it is more work than Variation C, which we like better.
 * Variation C: this is our ideal design, which is a modal over the context where the user came from.  It helps make it clear that user is close to returning to the task they were doing when they started creating an account. It also presents questions one at a time so the survey is not overwhelming.  This incorporates our learning from the comparative review.  We will have to design and engineer a separate mobile web version, as well as consider what no-Javascript users will receive.  This is being engineered during November and December 2018.

The team built and deployed Variation A in November 2018, and simultaneously built Variation C. We will be deploying Variation C during January 2019.

Questions, responses, and other text
The questions that this feature asks to newcomers, along with their specific wording and the other text in the feature, is really important to getting useful data. The team worked on these things with help from several other people experienced with surveys at WMF. All the questions will be optional, and will be translated into the languages of the wikis on which the welcome survey is deployed. Below are the questions we want to ask, and we are still working on the wording. To see the current text of the feature, check out the current mockups.


 * Why did you create your account today?
 * Goal: if we understand what a newcomer is trying to accomplish, we may be able to show them help materials that help them accomplish it.
 * Response options:
 * To fix a typo or error in a Wikipedia article
 * To add information to a Wikipedia article
 * To create a new Wikipedia article
 * To read Wikipedia
 * Other (please describe)
 * Have you ever edited Wikipedia?
 * Goal: we want to know how many people creating accounts may have already made IP edits. This would also allow us to refrain from giving too much help material to people who know what they're doing already.
 * Response options:
 * Yes, many times
 * Yes, once or twice
 * No, I didn't know I could edit Wikipedia
 * No, for other reasons
 * I don't remember
 * People can edit Wikipedia articles on topics they care about. We've listed a few topics below that are popular for editing.  Select some topics that you may wish to edit.
 * Goal: it may be possible in the future to connect newcomers with experienced editors who share their interests, or just give them recommendations on articles to work on.
 * Response options (see this update for explanation of how these were determined):
 * As checkboxes: Arts, Science, Geography, History, Music, Sports, Literature, Religion, Popular culture
 * Behind typeahead: Entertainment, Food and drink, Biography, Military, Economics, Technology, Film, Philosophy, Business, Politics, Government, Engineering, Crafts and hobbies, Games, Health, Social science, Transportation, Education
 * It will also possible for users to add their own topics.
 * Email address
 * Goal: Only if the user did not supply it on Special:CreateAccount. Email is important for engaging new editors and allowing them to recover their accounts, so we want to ask twice.
 * We are considering starting a program for more experienced editors to help newer users with editing. Are you interested in being contacted to get help with editing?
 * Goal: We want to learn whether newcomers feel that they need human-to-human help.

In addition to the questions themselves, we will be showing users a link to the survey privacy statement (translated into the wiki's language) and to the Tutorial and Help Desk pages on their wikis.

User testing the Welcome Survey
During the week of October 22, 2018, we used usertesting.com to conduct six tests of our Variation A mockups with internet users unaffiliated with the Wikimedia movement. In these tests, respondents are compensated for trying out the mockups, speaking aloud on what they observe, and answering questions about the experience. As our team's designer described on the Phabricator task, the goals of this testing were:


 * 1) Identify improvements to Understanding of the survey (update copy depending on users' comprehension of the instructions and questions on the form).
 * 2) Identify improvements to the Usability of the survey (check whether users are able to correctly input and submit their intended responses).
 * 3) * Do users know how the visual design and layout works? (Eg., do they understand they should be selecting one answer from a radio button group)
 * 4) * Are users able to navigate through the form?
 * 5) * Any feedback missing that could help users complete the form?
 * 6) Gauge user Reactions to the survey and Expectations of how the information will be used.

Summary of findings


 * Survey was clearly optional, and seen as short, low-effort, and non-intrusive to complete
 * Generally seen as data capture for research (though a few testers thought more specific explanation about how responses may be used might make them more inclined to fill it in)
 * "Mentorship program" was the one aspect not clearly understood, with about half of participants mistakenly assuming they would be providing the help rather than receiving it.
 * A couple of users had misgivings about providing email and feared it may be misused for marketing or given to 3rd parties (also as they expected it to be mandatory for account creation if it was needed for recovery)
 * Users liked the post-submission message with more information about "Getting started with editing"

Recommendations


 * Add a full Thanks message after survey completion
 * Rephrase “Mentor” checkbox question and clarify expectations for how users may be contacted if they select the mentor checkbox
 * Revise phrasing of Q3 (Wordsmithing so that the the second sentence “We've listed a few below popular for editing” is more easily understood.)
 * Add a tooltip beside or assistive text under the “Add more topics” field for no-js users explaining they can comma-separate multiple topics
 * Add more information about how the email is used
 * Potentially be extra clear by placing a tooltip with the following message (taken from Help:Account_management): “If you choose to give an email address, other users will be able to contact you by email. This feature is anonymous—the user who emails you will not know your email address. You don't have to give your email address if you don't want to (but doing so is required to reset your password if you forget it)”

Analysis and experiments
There are two kinds of analysis we'll be doing with respect to this feature. To read about those plans in-depth, see this page containing the "experiment plan".

Analysis of the responses to the survey
The main objective of the survey is to collect information about what new editors are trying to accomplish, so we can figure out if it's possible to personalize their experiences based on their responses. We are doing straightforward analysis on responses, broken out by elements like platform (mobile vs. desktop) and context from which the account was created (homepage, reading, editing).

The initial report on survey responses can be found here. Community members should feel free to translate this report into their languages. In-depth reporting will be created by March 2019. Some topline results:


 * The survey has high response rates: 67% in Czech and 62% in Korean.
 * Many people create accounts just to read articles -- a potential opportunity to engage new editors.
 * High numbers of people indicate they are interested in being contacted for help: 36% in Czech and 53% in Korean.

Analysis of the impact of the survey on user behavior
Although we would not have intended it, it is possible that the welcome survey could depress the number of newcomers who make edits. Community members first brought up this concern, saying that by making the sign-up process longer, and keeping newcomers away from their objectives, we could cause them to leave the site. Therefore, we conducted an A/B test to find out whether being presented with the survey decreases activation rate (the rate at which new users make their first edit).

From November 19 to January 15, half of newcomers in Czech and Korean Wikipedia were given Variation A of the survey, and half were given no survey. Our experiment results are published here, and show that having the survey does not significantly decrease the rate at which newcomers make their first edit. Based on this analysis, we are not testing Variation A against Variation C to see if Variation C has increased response rates. Results from this experiment are expected in February 2019. After this experiment is complete, we will determine whether Variation A or Variation C is the survey we should deploy to all newcomers in our target wikis going forward.

Part 2: Personalizing
In the Summary section above, we talk about how this project is divided in two parts. First, we are collecting information from new users about what they are trying to achieve and what they're interested in. We're doing that through the welcome survey. Second, we will use that information to personalize the experience they have on their first day, so that they can accomplish their goals and keep editing. Even though we've only just started collecting data from the welcome survey, our team is starting to talk about how to take action on that data. Our central question for Part 2 is: How might we personalize a newcomer's experience to help them achieve their goals?

In this section of this page, we're recording our ongoing thoughts and community discussion points around how to do this. Please participate here in the discussion.

Identifying newcomer goals
The New Editor Experiences research identified that most newcomers fit into. Those different personas have different skills and different goals, and we can increase retention if we develop an experience that is attuned to the persona of each editor. For instance, if we think a newcomer is a "", we might recommend a set of corrections to make. Our welcome survey questions are not granular enough to precisely classify newcomers into personas (that would require a much longer survey), but they give us some good information to work with. The table below shows an example of how we're thinking about this. Each row shows one of the six personas, along with a response from the welcome survey that might help us identify them, and a treatment that might go with that persona to help them be retained.

Potential interventions
To think through the potential ways to take action on the welcome survey responses, we went back through our team's original lists of potential interventions, several of which the community discussed on this page. These are three main approaches we could think of around how to personalize the first day using the welcome survey.

Approach 1: use responses to surface relevant content somewhere onwiki

There are four separate ideas inside this approach:


 * Task recommendations: surface tasks relevant to the reported goals, experience level, and topic interests of the newcomers.
 * Relevant help materials: surface help documents that are specific to the task the newcomer said they're trying to achieve, and the experience level they said they have.
 * Community visibility and topic routing: give newcomers visibility into activity and editors in the topics they care about, perhaps by allowing them to filter the Recent Changes feed by category, or by connecting them to WikiProjects. Perhaps by seeing that there is a vibrant community, they will be inspired to plug in and be active.
 * Newbie forum: create a place where newcomers can have visibility into the work done by other newcomers, along with help materials specifically geared toward being new.

Looking at these together, we realized that if we built any of these things, we would not have a good place to "put" them -- there isn't a page that a newcomer can go to to get resources specifically around being new. Perhaps pursuing this approach might mean building some kind of "newcomer homepage", where they can find and access these things. This idea actually came up in the user testing for the Welcome Survey, in which a couple users said they would expect to be taken to "their dashboard" after completing the survey.

Approach 2: mentorship program

If substantial numbers of newcomers respond affirmatively to the question about whether they would like to receive help from an experienced editor, that might be evidence that we should consider how to connect experienced and new editors in some sort of mentorship program. This is an idea that has been attempted multiple times in the Wikimedia movement, with varied success (e.g. Adopt-a-user in English Wikipedia). The of "human-to-human help" is also one of the strongest recommendations from the New Editors Experiences research. The challenge would be how to do this right, and how to rally a community around it.

Approach 3: engagement emails

This is an idea that was discussed specifically by the community in September 2018: send emails to new editors about their impact, or to remind them to make their first edits, or to otherwise encourage them to engage. The results from the welcome survey could help us make such emails more relevant to the user, and speak to the goals, experience level, or topics they indicated in the survey.