Growth/Positive reinforcement

This page describes work on "positive reinforcement" as part of the Growth feature set. This page contains major assets, designs, open questions, and decisions.

Most incremental updates on progress will be posted on the general Growth team updates page, with some large or detailed updates posted here.

Current status

 * 2021-03-01: project page created
 * 2022-02-25: project kicked off with team discussions
 * 2022-03-01: project page expanded
 * 2022-05-11: community discussion
 * 2022-08-12: user testing complete
 * Next: further design iteration & engineering starts on impact module improvements

Summary
The Growth team has been focused on building a “cohesive newcomer experience” that provides access newcomers need to the elements that help them join the Wikipedia community of practice. For instance, with newcomer tasks, we have given them access to opportunities to participate, and with the mentorship module, we have given them access to mentorship. Suggested edits has been able to get more newcomers to make their first edits. With that success, we want to take action to encourage newcomers to continue to make more edits. This draws our attention to an undeveloped element to which newcomers need access: evaluating performance. We’re calling this project “positive reinforcement”.

We want newcomers to understand there is progression and value to sustained contributions on Wikipedia, increasing retention for those users who took the first step in making an edit.

Our big question here is: How might we encourage newcomers who have visited our homepage and tried our features to keep editing and build on their momentum?

Background
When the newcomer homepage was deployed in 2019, it contained a basic "impact module", which listed the number of pageviews for the pages the newcomer had edited. That is the only part of the Growth features that give the newcomer any sense of their impact, and we have not improved on it since it was first deployed.

With this as a starting point, we have gathered some important learnings about positive reinforcement:


 * We have heard good feedback from community members about the module, with experienced editors saying that it is interesting and valuable to them.
 * Appreciation from other users has been shown to increase retention, such as in the case of "thanks" (here and [ https://citizensandtech.org/2020/06/effects-of-saying-thanks-on-wikipedia/ here]) and in an experiment on German Wikipedia. We believe that these reinforcements from real people would be more effective than automated ones coming from the system.
 * Community members have explained that it is a high priority for newcomers to move on to more valuable tasks after starting with easy ones, as opposed to getting stuck just doing easy tasks.
 * Other platforms, such as Google, Duolingo, and Github, all utilize numerous positive reinforcement mechanisms like badges and goals.
 * Communities are wary of incentivizing unhealthy editing. We have seen that when editing contests offer cash prizes, or just when useful roles such as "extended confirmed" rely on edit counts, it can incentivize people to make many problematic edits.

User persona


There are many parts of the newcomer journey in which we could attempt to increase retention. We could focus on newcomers who have stopped editing after just one or a few edits, or we could focus farther down the journey on newcomers who have stopped editing after weeks of activity. For this project, we have decided to focus on those newcomers who have completed their first editing session, and who we want to return for a second session. The diagram illustrates these with a yellow star.

We want to focus on newcomers at this stage, as that’s the next stage of the editor funnel in which we can help improve retention. It is also where we see a very significant attrition rate currently, so if we can help retain newcomers at this point, it should have a meaningful increase in editor growth overtime.

Research and design
Research was conducted on the various mechanisms that have been employed to encourage people to contribute content to both on and off-wiki products. The following are some of the key findings from the research:
 * Motivations for Wikipedia editors are multifaceted, and shift over time and experience. New editors are often driven more by curiosity and social connection than ideology.
 * Internal projects focus on intrinsic incentives, appeal to altruistic motivations, and are not systematically applied.
 * Broadening the appeal beyond ideological motivations may improve diversity of retained editors on Wikipedia.
 * Positive messages from experienced users and mentors is proven effective in short-term retention.

To see a summary of the current design ideas for Positive Reinforcement, see this [ https://docs.google.com/presentation/d/1MRRoUA7RNK1SVKdjY5U94YSnwzJ2tn6oKlja5-2C-hQ/edit#slide=id.p Design Brief].

Our designs will evolve further through community feedback and several rounds of user testing.

Ideas
We have three main ideas for positive reinforcement. We may pursue multiple ideas as we work on this project.

Impact

 * Impact: An overhaul of the Impact module based on incorporating stats, graphs, and other contribution information. The revised impact module would provide new editors more context about their impact, as well as encourages them to continue contributing.  Areas of exploration include:
 * Suggested edits milestone, to nudge users to try suggested edits.
 * Statistics on how much the user has edited over time (similar to what is in [ https://xtools.wmflabs.org/ec/en.wikipedia.org/Cloud%20atlas X Tools]).
 * “Thanks received” count, to highlight the ability to receive community recognition.
 * Recent editing activity - including days in a row newcomers have edited (“streaks”) to encourage continued engagement or remind people to restart their contributions.
 * View reading activity on articles newcomers have edited over time (similar to info on en:Wikipedia:Pageview_statistics).

Leveling up

 * Leveling up: It is important to communities that newcomers progress to more valuable tasks. For those who do many easy tasks, we want to nudge them toward trying more difficult tasks.  This could happen after they complete a certain number of easy tasks, or by encouragement on their homepage.  Areas of exploration include:
 * The newcomer will see success messages post-editing that motivate them to do more edits of the same or different levels of difficulty.
 * In the Suggested Edits module, provide opportunities to do more difficult edits, so that newcomers can become more skilled editors.
 * In the Impact module, include a milestone counter or award area.
 * On the Homepage, add a new module with set challenges to attain some reward (badge/certificate).
 * Add notifications to prompt newcomers to try a more difficult task.

Personalized praise

 * Personalized praise: research shows that praise and encouragement from other users increases newcomer retention. We want to think about how to encourage experienced users to thank and award newcomers for good contributions.  Perhaps mentors could be encouraged to do this on their mentor dashboards or through notifications.  We can utilize existing communication mechanisms which past studies have proven to have a degree of positive effect.  Areas of exploration include:
 * A personal message from the newcomer’s mentor appearing in the homepage.
 * An echo notification from the mentor or the Wikimedia Growth team.
 * “Thanks” on a specific edit.
 * A new milestone badge awarded by the mentor or the Wikimedia Growth team relating to a specific edit.

Community discussion
We discussed the Positive Reinforcement project with community members from ar:ويكيبيديا:مشروع فريق النمو (التعزيز الإيجابي)bn:উইকিপিডিয়া:আলোচনাসভাcs:Diskuse k Wikipedii:Zkušenosti nových wikipedistů/Pozitivní posílenífr:Discussion Projet:Aide et accueil/Volontaires, and here on mediawiki.org.

We received direct feedback about the three main ideas, along with many other ideas for improving new editor retention.

Below is a summary of the main themes from the feedback, along with how we plan to iterate based on the feedback.

Other ideas:
Community members suggested several other ideas for improving newcomer engagement and retention. We think these are all valuable ideas (some of which we are already exploring or want to work on in the future) but the following ideas won't fit within the scope of the current project:
 * Send newcomers onboarding and welcome emails (the Growth team is actually currently exploring engagement emails in collaboration with the Marketing and the Fundraising teams).
 * Expose newcomers to Wikiprojects that relate to their interests.
 * Include a customizable widget on the newcomer homepage to allow wikis to promote certain newcomer tasks or events.
 * Send notifications to users who welcome newcomers once the newcomer reaches certain editing milestones (to help prompt the user to offer Thanks or Wikilove).

User testing
Along with community discussion, we wanted to validate and add to our initial designs and hypothesis by testing designs with readers and editors from several countries. So our design research team conducted Positive Reinforcement user testing aimed to better understand the project's impact on newcomer contribution across several different languages.

We tested several static Positive Reinforcement designs with Wikipedia readers and editors in Arabic, Spanish, and English. Along with testing Positive Reinforcement designs we introduced data visualizations from xtools as a way to better understand how these data visualizations are perceived by newcomers.



User testing results

 * Make impact data actionable: Impact data was a compelling feature for participants with more experience editing, which several related to their interest in data—an unsurprising quality for a Wikipedian. For those new to editing, impact data, beyond views and basic editing activity, may be more compelling if linked to goal-setting and optimizing impact.
 * Evaluate the ideal editing interval: Across features, daily intervals seemed likely to be overly ambitious for new and casual editors. Participants also reflected on ignoring similar mechanisms on other platforms when they were unrealistic. Consider consulting usage analytics to identify “natural” intervals for new and casual editors to make goals more attainable.
 * Ensure credibility of assessments: Novice editor participants were interested in the assurance of their skills and progress the quality score, article assessment, and badges offer. Some hoped that badges could lend credibility to their work reviewed by more experienced editors. With that potential, it could be valuable to evaluate that the assessments are meaningful measures of skill and further explore how best to leverage them to garner community trust of newcomers.
 * Reward quality and collaboration over quantity: Both editor and reader participants from esWiki were more interested in recognition of their knowledge or expertise (quality) than the number of edits they have made (quantity). Similarly, some Arabic and English editors are motivated by their professional interests and skill development to edit. Orienting goals and rewards to other indicators of skilled edits, such as adding references or topical contributions, and collaboration or community involvement may also help mitigate concerns about competition overtaking collaboration.
 * Prioritize human recognition: While scores and badges via Growth tasks is potentially valued, recognition from other editors appears to be more motivational. Features which promote giving, receiving, and revisiting thanks seemed most compelling, and editors may benefit from selecting impact data which demonstrates engagement with readers or editors most compelling to them.
 * Experiment with playfulness of designs: While some positive reinforcement features can be seen as the product of “gamification”, some participants (primarily from EsWiki) felt that simple, fun designs were overly childish or playful for the seriousness of Wikipedia. Consider experimenting with visual designs that vary in levels of playfulness to evaluate broader reactions to “fun” on Wikipedia.

Design


Below are the current designs for Positive Reinforcement. We have refined the three main ideas outlined above, but the scope of plans and the actual designs have evolved based on feedback from community discussions and user testing.

Impact
Phabricator epic: T222310

The revised impact module provides new editors with more context about their impact. The new design includes far more personalized info and data visualizations than the previous design. This new design is fairly similar to when discussing this feature with communities. You can view the current engineering progress at beta wiki, and we hope to release this feature to Growth pilot wikis soon.

Leveling up
Phabricator epic: T317813

The Leveling up features focus on encouraging newcomers to progress to more valuable tasks. Ideas also include some prompts for new editors to try suggested edits, since structured tasks have been shown to improve newcomer activation and retention.


 * “Level up” post-edit dialog message: A new post-edit dialog message type is added to encourage newcomers to try a new task type. We hope this will encourage some users to learn new editing skills as they progress to different, more challenging tasks.
 * Post-edit dialog for non-suggested edits: Introduce newcomers who complete ‘normal’ edits to suggested edits. We plan to experiment by showing newcomers a prompt post 3rd and 7th edit. Desktop users who click through to try a suggested edit will also see their Impact module, which we hope helps engage newcomers and provides a small degree of automated positive reinforcement. We will carefully measure this experiment, and ensure there aren't any unintentional negative effects.
 * New notifications: New echo notifications to encourage newcomers to start or continue suggested edits. This acts as a proxy to “win-back” emails for those who have an email address and settings on to receive email notifications.

Personalized praise
Phabricator epic: T317817

Personalized praise features are based on research results that show that encouragement and thanks from other users increases editor retention.


 * Encouragement from Mentors: We will add a new module to the Mentor dashboard, that is designed to encourage Mentors to send personalized messages to newcomers who meet certain criteria. We will allow Mentors to customize and control how and when "praise-worthy" mentees are surfaced.
 * Increasing Thanks across the wiki: We plan to fulfill the community wishlist item to Enable Thanks Button by default in Watchlists and Recent Changes. We hope this will increase Thanks and positivity across the wikis, and hopefully newcomers will benefit from this directly or indirectly.

Measurement and results
Once community discussion is complete, designs are refined, development and testing are complete, our staff Data Scientist will closely monitor the impact of the Positive Reinforcement project. We will share our initial measure plan and subsequent results here.