Recommendations to support language onboarding are formulated through a series of conversations with various stakeholders from WMF staff and the volunteer community. See below.
Next hypothesis - 2.2.6: If we document the pre-incubator, incubator, and post-incubator journeys for the five pilot wikis with quantitative and qualitative data, we will be able to better support new languages in the future.
For the qualitative analysis plan with active editors, alongside the quantitative analysis, please get in touch with Wikimedia_Research.
As an onboarding resource, the Starter_kit can be used and and support from the Language Diversity Hub and Telegram can help us stay in contact with the new wiki communities throughout the pilot.
The meeting focused on preparations for setting up wikis for the five newly approved languages, discussing timelines, necessary features, and monitoring strategies. The team also highlighted the importance of documenting the wikis' journeys and maintaining communication with new wiki communities.
Minimum 2 active editors in the last three months sounds good. Exclude editors editing across more than 5 languages.
What would be considered a success to make this intervention "worth it"?
Does it meet the Language Committee's criteria for graduation? If not, how does it compare to the control group?
Establish a percentage range to distinguish between 'average' and 'above average' outcomes.
Explore developing a campaign toolkit for how-to use the features/tools that are not yet on Incubator and list of articles new wikis should have using the following resource as a reference.
In the meeting, the measurement plan was reviewed along with selection criteria in the Language Committee proposal and the success criteria for the experiment.
Experiment Hypothesis: If we provide production wiki access to incubating language projects, this will result in at least X% additional editor productivity, leading to X% additional content generation than usually on incubator wiki.
Baselines:
Look at averages.
Examine activity when projects "graduate" from the Incubator to a production wiki.
Compare similar wikis to avoid comparing dissimilar ones.
Set Percentages: Abandon a set percentage, OR consult SRE for successful intervention criteria.
Effect Measurement:
Project Level:
Net content.
Pages (content, template, and rest).
Bytes (content).
Average number of monthly active editors (active: at least 5 constructive edits).
Editor Level (Productivity):
Average number of constructive edits.
Editing session completion rate.
Tool usage.
Content Translation: Determine the time required to enable content translation for a new language project (TODO: Consult Language team).
Visual editor: Available on Incubator but still buggy.
Citoid, Wikidata.
Extensions: Complete list of tools/extensions/features not on Incubator available here.
Interlanguage links and other additions not available on Incubator.
Future Hypothesis: How to support tool usage if there is no uptick.
Timeline:
Likely 1 month for editing in this pilot.
Initial analysis and follow-up analysis.
Evaluate in 1 month, but unlikely to prove or disprove anything.
Will not stop intervention unless there is vandalism or similar issues.
Determining Success:
Success if compelling evidence shows it's working, with input from LangCom.
If no compelling case, extend time and create additional hypothesis next quarter.
Continue study based on learnings.
Onboarding Support:
General onboarding support should be aligned with the Language Inclusion Support Plan.
Minimal onboarding to reduce bias in measurement.
Provide how-to resources equally to all 5 wikis for new features on a production wiki. Resources:
How to use Content Translation, Visual editor, citation tool, etc.
Link to existing resources on wiki.
Action items
Look at one-month and three-month averages for editing baselines
Look in to which features/tools the production wiki will offer that aren’t available on Incubator
Look into what resources exist for how-to use these ^ features/tools
Find out from SRE and others what would be considered successful to make this intervention "worth it"
The team brainstormed criteria for measuring the success of the incubation pilot. The hypothesis is that providing production wiki access to incubating language projects will increase editor productivity and content generation by at least X%. Baselines will be identified by examining averages and activity during project graduations from the Incubator to a production wiki. The discussion included measuring effects at both project and editor levels, and providing equitable onboarding support. Action items include reviewing editing baselines, identifying additional tools not available in Incubator, finding related resources, and consulting SRE on success criteria.
Language Onboarding and Development/Community conversations 2024-25
