Growth/Growth team updates/2022

Update 2022-09-28: Mentorship improvements and increase in “add a link” confidence score

 * Mentorship
 * The new Structured mentor list is now deployed at Arabic, Bengali, Czech, and Spanish Wikipedias (T310905) / (T264343). After a period of testing, feedback, and some further improvements, we will release this change to all wikis. The new structured mentor list has several advantages:
 * Easier set up: Previously each community that wanted to adopt Growth mentorship had to create its own mentor list. With the structured mentor list, the only step needed to get Growth mentorship will be to get a sufficient number of mentors on board.
 * Better extendibility: The new system allows mentors to easily update their mentor settings, and allows the Growth team to further improve and extend mentor features.
 * Less error-prone: One user’s “bad” edit can no longer break a list or enroll the wrong user.
 * Easier to parse: Among other improvements this will allow the Growth team to provide more meaningful data about mentorship at each wiki. Please let us know in this phabricator task or on a Growth team talk page if you have any feedback about what mentor or mentee data you want to have for your community.
 * We have several Mentorship improvements in the works, including:
 * When a mentor gets blocked, mark them as "Away" (T318819)
 * Show information about mentees' pending changes on FlaggedRevs wikis (T297811)
 * Change the default values to become a mentor (T318545). Communities can configure this based on their preferences, but for wikis that don't customize, the default will be:
 * Minimum number of days users should be registered to become a mentor : 90
 * Minimum number of edits users need to become a mentor : 500


 * Add a link
 * The Growth team has increased the confidence score on link recommendations (T316079). This means that all future link suggestions added to the Suggested Edits queue are more likely to be accurate. It’s important to note that this change will ensure that more of the “add a link” suggestions are correct, but there will still be link suggestions that are incorrect or unnecessary, which is why human review of these suggestions is necessary. All newcomers who complete this task receive onboarding that explains: Suggested links are machine-generated, and can be incorrect. The suggestions might be on words that don’t need them, or might link to the wrong article. Use your judgment to decide whether they are right or wrong.

Update 2022-09-20: Copy edit research, “add an image” scaling, and structured mentor list release

 * Copy edit structured task
 * The Research team and the Growth team are continuing to investigate the best strategy for introducing a copy edit structured task that would guide newcomers through the process of making a spelling or grammar fix. We just completed an initial evaluation of two different tools across 5 different languages: Arabic, Bengali, Czech, Spanish, and English. You can read about our results here.
 * Add an image
 * The Growth team deployed the “add an image '' suggested edit to 5 more Wikipedias (T314518) on Monday, September 19, 2022. We have communicated with these wikis (T301601), and we will continue to listen to feedback about this newcomer task. The 5 wikis that received "add an image" this week are:
 * Greek Wikipedia (el)
 * Polish Wikipedia (pl)
 * Chinese Wikipedia (zh)
 * Indonesian Wikipedia (id)
 * Romanian Wikipedia (ro)
 * Mentorship
 * This Wednesday, September 21, 2022, a new structured sign-up process will be released for Mentorship to three Growth pilot wikis: Arabic, Bengali, and Czech Wikipedias (Growth Mentorship tools haven’t yet been enabled by the Spanish Wikipedia community). The Structured mentor list is a new way for mentors to sign up. Instead of signing up via a wikitext-based list, mentors will soon be able to use a form to enroll and then utilize the Mentor dashboard to update information, such as their welcome message or number of newcomers they want to be assigned to.  Learn more about this project here: Structured mentor list.

Update 2022–09-14: Onboarding improvements and instrumenting blocked account registration

 * Onboarding improvements for new editors:
 * We continue to carefully measure how Growth tools impact new users across wikis. Although Growth features improve newcomer engagement as a whole, we noticed that the current onboarding wasn’t working as well for newcomers who start their first edit, and then create an account before publishing the edit. We’ve made improvements to help better accommodate this user journey (T310320).
 * Instrument blocked account registration
 * There is concern around the impact that large IP blocks are having on newcomers (wikimedia-l thread). Wide IP ranges that are associated with open proxy services are sometimes blocked to stop persistent sockpuppet accounts and vandals from avoiding blocks. But this IP blocking has the negative side effect of blocking many good-faith newcomers from creating accounts and editing. The Growth team has completed a task to log an event when account creation is denied so that we will have more complete data around account blocks related to local IP blocks, local autoblocks, and global IP blocks (T306018).

Update 2022–09-05: Newcomer & Mentorship data

 * Growth KPIs
 * We’ve added more data to the Growth KPIs, so you can better understand how Growth tools are utilized on your wiki. If you have any questions, or there is more data you want access to, please let us know.
 * Clicks / Saves / Reverts: You can not only view the total number of each Suggested Edit task type that is clicked on and completed on your wiki, but also the revert counts for each task type.
 * Questions posted to mentors or help desk: You can view the number of questions posted to mentors, along with where those questions originate.
 * Account registration: You can view the number of new accounts created on your wiki and see if there is an experiment running resulting in a certain percentage of newcomers not automatically receiving Growth features.
 * Newcomer task edit type analysis
 * Communities had expressed concern that newcomers whose initial edits were structured tasks wouldn’t go on to learn how to complete more difficult tasks, so a WMF data scientist conducted Newcomer task edit type analysis to see if this was indeed the case. Results from analysis indicate that this likely isn’t a significant concern. In this analysis, we use data from wikis where Add a Link was deployed and examine whether newcomers who start out with Add a Link edits end up continuing to only make those types of article edits. We also do the same for users who started out with a different Newcomer Tasks edit, so we can compare between the two groups. We find that both groups are more likely to go on to make additional article edits compared to those who do not start out with Newcomer Tasks, and that more than 70% of newcomers who start out with Add a Link go on to make a combination of Newcomer Tasks and other article edits or only other kinds of article edits. Read the full analysis and methodology here.

Update 2022-08-22: Copy Edit analysis and Structured mentor list

 * Copy Edit structured task
 * The Research team created a list of 100 samples of potential copy edits in Wikipedia articles for arwiki, bnwiki, cswiki, eswiki (Growth pilot-wikis) and enwiki with two different methods: LanguageTool and Hunspell. LanguageTool is a free and open-source grammar, style, and spell-checker. Hunspell is a free spell checker and morphological analyzer used by LibreOffice, OpenOffice, and many other open source projects. The Growth team Ambassadors will review these copy edit samples to determine the level of suggestion accuracy for each tool using actual Wikipedia articles.  We will share more details here and on the associated Copy Edit page once the evaluation is complete.
 * Mentorship
 * The Growth team hopes to release the new Structured mentor list to Growth pilot wikis in September. We will work to ensure the Structured mentor list documentation is translated and then connect with Arabic, Bengali, Czech, and Spanish communities about the change to ensure the transition goes smoothly for current and future mentors. If you want to know what to expect, the change is already testable at all beta wikis and at test.wikipedia.org. You can view Phabricator task T312102 for more details.

Update 2022-08-17: Copy Edit research, Structured mentor list development, and Positive Reinforcement user testing

 * Copy Edit structured task
 * In the future, the Growth team plans to develop a Copy Edit structured task: a task related to improving article spelling, grammar, punctuation, tone, etc. In our initial community discussions around structured tasks, we found that copyediting was the task that communities desired most. There are many challenges and unknowns associated with supporting so many languages, so we are working with the Research team on reviewing our various options.  We have published initial Research findings here. We are now working towards building a list with examples of automatically suggested copy edits for manual evaluation by the Growth Ambassadors (T315086).
 * Mentorship
 * A new structured sign-up process will be released for Mentorship soon. The Structured mentor list is a new way for mentors to sign up. Instead of signing up via a wikitext-based list, mentors will soon be able to use forms incorporated into the Mentor dashboard to enroll into mentorship and to change their information, such as their message or number of newcomers they want to be assigned to. This update is already testable, please view this Phabricator task if you want to test the changes.  Learn more about this project here: Structured mentor list.


 * Positive Reinforcement
 * Our design research team completed Positive Reinforcement user testing which aimed to better understand the project's impact on newcomers across several different languages. We are now utilizing user testing feedback along with prior community feedback to iterate and improve designs.

Update 2022-08-08: Add a Link analysis and Positive Reinforcement user testing

 * Add a link
 * This week we will finish analyzing the impact of prioritizing underlinked articles for “add a link” (T301096). The initial development work is complete, now we are testing results compared with the current baseline to ensure the new method of prioritization is an improvement.
 * When an "add a link" suggestion is rejected by a newcomer, they are asked to select a reason why. Analysis found that "Other" was chosen as the rejection reason for link rejections about 14% of the time. We temporarily gathered free text rejection reasons to help us identify if there is a rejection reason we should add or an improvement to make to the algorithm (T304099 & T313141). We are analyzing responses this week and hope to have a decision made soon.
 * Positive Reinforcement
 * User testing of initial Positive Reinforcement designs was just completed. Interviews were conducted in Arabic, English, and Spanish. We will share this data on the Positive Reinforcement page soon.
 * We are continuing Impact Module work for the upcoming positive reinforcement project (T313310). The new Impact Module will include a similar overview of edited articles like the current design, but also include other data points and data visualizations to help newcomers understand the value and impact of their contributions.

Update 2022-07-25: Positive Reinforcement development starts

 * Positive Reinforcement
 * We have completed two engineering tasks to kick-off the Positive Reinforcement project. We have a configuration flag set up that will allow for progressive rollout of the new Impact Module (T311299). And we have completed the initial technical research and discussions needed to decide how to handle the backend storage and API for the additional data that will be added to the Impact Module (T309747).
 * We are wrapping up user testing for Positive Reinforcement prototypes this week (T307950), and then we will focus on finalizing designs based this user testing feedback along with previous community feedback (T313271).
 * Mentorship
 * A new structured Mentor sign-up process will be released soon. This new sign up method is already testable on beta Wikipedia, and we welcome any feedback (T312102). We will share documentation and details on our rollout strategy soon.
 * Mentors will see a new Mentee overview soon. This is part of the work our team is completing to migrate the Mentor Dashboard to Vue.js. There should be no feature changes, but this work should move the Growth features closer towards the Design System Team's vision for less feature redundancy, more consistent experience across projects, better accessibility and localization support, and the ability to move faster in the future (T300532).
 * Add a Link
 * We have been working on a task to better prioritize articles that are underlinked. We have a path forward, but first we are validating that the new sorting method is indeed valuable for our communities. Growth team ambassadors will compare before and after results to evaluate the impact of the change (T301096).

Update 2022-07-11: Mentorship improvements

 * Mentorship
 * To make it easy for mentors to sign up without accidentally breaking the list of mentors, the Growth team is working on a sign up page for mentors. Instead of the current wikitext-based mentor list, a structured mentor list will be created. This new structured list, called MediaWiki:GrowthMentors.json, will be the new canonical location for the list of mentors. To ensure this new process is well tested, we have rolled it out on beta English Wikipedia. If you are interested in testing and providing feedback, please see Phabricator task T312102 for testing instructions.
 * After the structured mentor sign up task is complete, it should be easier for us to complete a long standing community request to automatically reassign mentees when a mentor quits mentorship or is removed from mentorship by an Admin (T272376).
 * The structured mentor sign up process will also allow us to validate the maximum length of a mentor message. This should help ensure that a Mentor’s introductory message to mentees is readable and doesn’t get cut-off (T312576).
 * Add an image
 * We have moved from the proof-of-concept image suggestions API, to the new productionized version of the API (T306032). There should be no user-facing changes, but this ensures the future support and scalability of the “add an image” structured task.

Update 2022-07-05: Metrics and improvements for structured tasks

 * Growth team metrics
 * As of the last week of June 2022, newcomers have completed over 500,000 suggested edits from the newcomer homepage!
 * Some of the key performance indicators the Growth team tracks are now available in Grafana: Growth Team Product KPIs.
 * Add an image
 * We resolved a bug with “add an image” (T311916) and we are now working to improve end-to-end tests for "add an image" (T310351).
 * The Growth team has been gathering patroller feedback about “add a link” and “add an image” structured tasks from Spanish, Arabic, and Russian Wikipedias. Using this feedback, we are planning improvements to ensure structured tasks aren't a burden to patrol.
 * We are moving to a new Image Suggestions API (T306032).
 * Add a link
 * We are working on a task to better prioritize underlinked articles for “add a link” (T301096).
 * Positive Reinforcement
 * User testing of initial positive reinforcement designs is underway. Interviews are being conducted in Arabic, English, and Spanish.
 * We have started the initial Impact Module work for the upcoming positive reinforcement project (T311299).

Update 2022-06-27: Add a link analysis

 * Add a link
 * Add a Link Experiment Analysis has been published. The most important points are:
 * Newcomers who get the Add a Link structured task are more likely to be activated (i.e. make a constructive first article edit).
 * They are also more likely to be retained (i.e. come back and make another constructive article edit on a different day).
 * The feature also increases edit volume (i.e. the number of constructive edits made across the first couple weeks), while at the same time improving edit quality (i.e. the likelihood that the newcomer's edits are reverted).
 * A fourth round of wikis have received "add a link" (T304548).
 * This round included: Abkhazian Wikipedia, Achinese Wikipedia, Adyghe Wikipedia, Afrikaans Wikipedia, Akan Wikipedia, Alemannisch Wikipedia, Amharic Wikipedia, Aragonese Wikipedia anwiki, Old English Wikipedia, Aramaic Wikipedia, Egyptian Arabic Wikipedia, Asturian Wikipedia, Atikamekw Wikipedia, Avaric Wikipedia, Aymara Wikipedia, Azerbaijani Wikipedia, and South Azerbaijani Wikipedia.
 * Add an image
 * This feature was utilized at GLAM events in Argentina, Mexico, and Chile. For an overview of what was learned from these events, read: #1Pic1Article I: how Latin American heritage experts added images to Wikipedia.

Update 2022-06-13: Structured tasks and positive reinforcement feedback

 * Structured Tasks
 * We have been discussing the challenges of patrolling "add a link" and "add an image" structured tasks with patrollers on Arabic Wikipedia, Spanish Wikipedia, and Russian Wikipedia. We are now brainstorming improvements to help address some of the main patroller complaints. We would love to have feedback from an even wider audience, so if you have any thoughts to add about the challenges of reviewing these tasks or how we should improve these tasks further, please let us know.


 * Positive Reinforcement
 * We summarized the initial feedback gathered from community discussions about positive reinforcement ideas, along with how we plan to iterate based on that feedback. The first Positive Reinforcement idea (an overhaul of the impact module based on incorporating stats, graphs, and other contribution information) received the widest support and we plan to move forward with impact module design A.

Update 2022-06-06: Community configuration for "add a link" and "add an image"

 * Add a link
 * Communities can set the maximum number of "add a link" suggested tasks a newcomer can complete daily. In other words, communities can limit the number of "add a link" tasks newcomers can complete, which is helpful if patrollers are overwhelmed by the quantity of these edits. Privileged users can configure this task limit, and other Growth feature settings, from Special:EditGrowthConfig. (T308543).
 * The backend work is complete for the 4th round of wikis to receive "add a link" (T304548).
 * We posted an overview of what is included in "add a link" iteration 2.
 * Add an image
 * A bug was fixed in which the "view image details" option was no longer visible when a newcomer was adding a caption.
 * Communities can set the maximum number of "add an image" suggested tasks a newcomer can complete daily (T308543).
 * Positive reinforcement
 * We are continuing discussions with communities in Arabic, Bengali, Czech, English, and French, and starting to compile a summary we will share.
 * User testing of positive reinforcement designs will start soon. Tests will be conducted in Arabic, English, and Spanish.

Update 2022-05-30: positive reinforcement discussion, mentorship update, and "add a link" improvements

 * Positive reinforcement
 * We have started community discussions about the positive reinforcement project in Arabic, Bengali, Czech, English, and French. We would love to hear from more people, so please join one of these discussions if you are interested.
 * Based on community feedback, we added a few more designs to the "leveling up" idea. You can see one of our ideas in the accompanying image. Please join the discussion to share your thoughts.


 * Mentorship module:
 * We completed some improvements to the newcomer homepage that allows newcomers to better understand the mentorship feature, and allow them to opt out of mentorship.
 * Provide more information about the public nature of the mentors and talk pages in the Mentor module (T292627).
 * Allow newcomers to opt-in and opt-out of mentorship (T287915).


 * Add a link
 * The next group of wikis that will receive "add a link" will be:
 * Abkhazian Wikipedia
 * Achinese Wikipedia
 * Adyghe Wikipedia
 * Afrikaans Wikipedia
 * Akan Wikipedia
 * Alemannisch Wikipedia
 * Amharic Wikipedia
 * Aragonese Wikipedia
 * Old English Wikipedia
 * Aramaic Wikipedia
 * Egyptian Arabic Wikipedia
 * Asturian Wikipedia
 * Atikamekw Wikipedia
 * Avaric Wikipedia
 * Aymara Wikipedia
 * Azerbaijani Wikipedia
 * South Azerbaijani Wikipedia
 * We have tested and evaluated the models used for "add a link" for all of these languages. To stay informed, you can subscribe to Phabricator Task 304548.
 * We are working on two more "Add a link" improvements:
 * Prioritize suggestions of underlinked articles (T301096).
 * Allow communities to customize the "add a link" quality gate threshold easily within Special:EditGrowthConfig (T308543).
 * In our "add a link" analysis we found that this feature helps engage newcomers and has a low reversion rate. However, the ease of the task also means that on some wikis there are many of these newcomer edits appearing for patroller review. We are engaged in discussions with community members from several wikis regarding improvements we could make to the "add a link" feature or even the "Recent Changes" feed to ensure the feature is sustainable for patrollers.

Update 2022-05-23: positive reinforcement updates and other Growth team work

 * Annual planning
 * We are planning for our coming year, and we’ll post our thoughts in the next couple weeks so that community members can react to our plans and help make sure we’re on the right track.


 * Positive reinforcement
 * This is the next big project that the Growth team will focus on. We want newcomers to understand there is progression and value to sustained contributions on Wikipedia, and we want to improve new editor retention.
 * We have started a Positive Reinforcement discussion with contributors in English, and we will also discuss the project in more detail with our pilot wikis (Arabic, Czech, and Bengali Wikipedias) who receive Growth team features first. You can see some of our ideas in the accompanying image.  Please join the discussion to share your thoughts!
 * The three main ideas part of the Positive Reinforcement project are:
 * Impact: An overhaul of the Impact module based on incorporating stats, graphs, and other contribution information.
 * Leveling up: It is important to communities that newcomers progress to more valuable tasks. For those who do many easy tasks, we want to nudge them toward trying more difficult tasks.
 * Personalized praise: Research shows that praise and encouragement from other users increases newcomer retention. We want to think about how to encourage experienced users to thank and award newcomers for good contributions.


 * Add an image
 * There have been no changes lately to this feature, but we are gathering data and community thoughts to plan iteration 2.  So far, over 5,000 images have been added to articles.
 * Add an image was used in four successful GLAM events, in which 67 people made 198 image edits.  We also learned many of these contributors returned and continued to make edits following the GLAM event.
 * Add a link
 * We have completed "iteration 2", which involved several improvements that came from community discussion and from data analysis:
 * Algorithm improvements:
 * Avoid recommending links in sections that usually don't have links.
 * Avoid suggesting first names.
 * Limit each article to only having three link suggestions by default (limited to the highest accuracy suggestions of all the available ones in the article.)
 * User experience improvements:
 * Add a confirmation dialog when exiting out of suggestion mode prior to making changes.
 * Improved post-edit dialog experience.
 * Allow newcomers to browse through task suggestions from the post-edit dialog.
 * Community configuration:
 * Allow communities to set a maximum number of links per article via Special:EditGrowthConfig
 * With those improvements, we are confident about releasing "add a link" to more wikis, and so this past week the feature was released to these wikis:
 * Catalan Wikipedia
 * Hebrew Wikipedia
 * Hindi Wikipedia
 * Korean Wikipedia
 * Norwegian Bokmål Wikipedia
 * Portuguese Wikipedia
 * Simple English Wikipedia
 * Swedish Wikipedia
 * Ukrainian Wikipedia
 * The next group of wikis that will receive "add a link" are documented in this this Phabricator task. The list may change slightly as we test and evaluate models and ensure the level of precision and recall in each language is adequate.
 * Marketing:
 * We are currently partnering with the Marketing team to test the efficacy of social media ads (bringing users into our projects) and welcoming emails (retaining new users). This newcomer experience marketing pilot includes an inspiring video full of community members.  We’ll post results in a few weeks.
 * Fundraising:
 * We are working with Fundraising on ideas for how to give donors the opportunity to learn to edit.  Next week we will start an experiment (in fundraising campaigns in Latin America, South Africa, and India) that shows donors a thank you message that also encourages them to create an account to start editing and improving Wikipedia.
 * Donors included in this fundraising campaign who create accounts will then be brought to the newcomer homepage to get started.

Between January and May 2022, some team members transitioned to new positions and responsibilities. As a consequence, only the main events have been written down below.

Update 2022-02-25: positive reinforcement project beginning

 * Add an image
 * This feature was originally deployed on mobile on 2021-11-29. On 2022-01-28, we deployed a desktop version so that newcomers can use it on either platform.
 * On 2022-02-16, we added Spanish Wikipedia to the test. Now, 40% of newcomers on Spanish, Arabic, Bengali, and Czech Wikipedias are receiving "add an image" as their first task.
 * Usage of the feature remains low, but with some users completing many edits day after day. We are now analyzing the full conversion funnel in detail to understand how to increase participation.  We are also analyzing the captions users write to understand their quality and how those can be improved.
 * This feature will be used in a GLAM event by Wikimedia Argentina on March 7, to help museum professional illustrate articles in Spanish Wikipedia. If it goes well, it will be used in several subsequent events in Latin America.
 * Add a link
 * The first iteration of "add a link" was deployed on 2021-05-29, and it is now on 11 wikis. Our data analysis shows that this feature has a substantially positive impact on constructive article activation -- it increases the likelihood that a newcomer makes an unreverted edit to an article by 17% (details forthcoming).
 * In the past months, we have collected community feedback, usage data, and user feedback. We are now taking this information to make a series of improvements that we're calling "iteration 2".  After making these improvements, we'll plan to ask more Wikipedias to take on this feature.  The improvements underway include:
 * Avoid recommending links in certain sections
 * Recommend a maximum of three links per article
 * Giving users more options of what to do after completing an edit
 * We'll also be experimenting with whether having several screens of onboarding does, in fact, help newcomers complete the task correctly
 * Positive reinforcement
 * In building suggested edits, the Growth team has successfully caused more newcomers to make their first edits to Wikipedia. But our analysis shows that although the features increase how many people make their first edits, they do not additionally increase the share who come back and edit again another day (retention).  For this reason, we are prioritizing a project called "positive reinforcement", which is about helping newcomers be proud of their impact and be excited about coming back to continue their editing.
 * This project has three main ideas, and we'll begin conversations about them with communities:
 * Impact: the newcomer homepage contains a rudimentary impact module that we could improve to make more interesting.
 * Leveling up: it is important to communities that newcomers progress to more valuable tasks. For those who do many easy tasks, we want to nudge them toward trying more difficult tasks.
 * Personalized praise: research shows that praise and encouragement from other users increases newcomer retention. We want to think about how to encourage experienced users to thank and award newcomers for good contributions.