Wikimedia Technical Documentation Team/Doc metrics
Please do not mark this page for translation yet. It is still being drafted, or it contains incomplete translation markup which should be fixed before marking for translation. |
Documentation health metrics for MediaWiki
Documentation metrics to help measure key aspects of the developer experience.
|
Measuring the health of our technical documentation can help us assess the developer experience and the state of the Wikimedia technical ecosystem. This project will identify which metrics are relevant for MediaWiki documentation, how we can use those metrics to drive our technical documentation work, and how they can help us identify areas for improvement.
Project overview
[edit]As part of the 24/25 annual plan, the Technical Documentation team's goal is to identify and establish metrics that measure the health of Wikimedia technical documentation, using MediaWiki Core documentation as a test case.
We want to measure doc health because documentation is a core component of the developer experience. This project seeks to explore and define:
- What questions in the field of developer/engineering workflows (see WE6) can documentation metrics help us answer?
- What type of documentation data can help enable informed decision-making about how to improve developers' experience?
One expected outcome of this work is a clearer definition and shared understanding of what it means to drive tech docs work with data in the Wikimedia context, and how we can start to build a measurement approach that we can apply to docs in various Wikimedia technical domains (i.e. beyond MediaWiki core).
Goals
[edit]- Define an initial set of documentation health metrics for MediaWiki core technical documentation.
- Implement the metrics for a pilot subset of MediaWiki core docs.
Timeline overview
[edit]Timeframe | Phase | Description |
---|---|---|
July– | Research | Doc metrics and available data |
August–September 2024 | Design | Draft v0 metrics definitions |
September– | Research/design | Doc collections and content scope |
September– | Implement | Doc collections for test |
Test | Measurement of doc collections for test of v0 metrics | |
November - December 2024 | Evaluate | Measurement of doc collections for test of v0 metrics |
Decide | Determine whether to proceed with gathering community feedback and metrics implementation |
Project milestones
[edit]The team plans to complete a research phase to help focus and scope our work, followed by a design and implementation phase where we generate and start using doc health metrics for a pilot collection of technical content.
Define a draft set of v0 metrics definitions for MediaWiki core technical documentation
[edit]Done phab:T372102.
Phase | Description | Timeframe |
---|---|---|
Research | Defined the dimensions of tech docs quality that are relevant for MediaWiki docs. Resolved RQ1. | By August 31 |
Research | Identified data signals that could help us measure those dimensions. Resolved RQ2. | By August 31 |
Research & Design | Mapped available data signals to quality dimensions. Identified signals that may require support from other teams, or extra work to implement. | By September 15 |
Design | Decided on and defined a set of v0 proposed metrics to explore with sample doc collections. | By September 15 |
Identify doc collections to measure
[edit]Done phab:T374722.
Phase | Description | Timeframe |
---|---|---|
Research | Define the scope of what to measure. Resolve RQ3. | By October 15 |
Design | Use either developer workflows, MediaWiki core code structure, or some other logical framework to identify several content collections to measure. | By October 30 |
Implement | Use PagePile or other mechanism to create collections. | By October 30 |
Test v0 metrics on doc collections
[edit]In progress phab:T379431
Phase | Description | Timeframe |
---|---|---|
Implement/Test | Test measurement of doc collections using v0 metrics (likely manual assessment). | By Nov 15 |
Evaluate | Use of v0 metrics on sample doc collections. | By Dec 30 |
Get feedback and community input on v0 metrics and doc collections
[edit]Phase | Description | Timeframe |
---|---|---|
Outreach | Present v0 metrics proposal for community feedback. Explain the concept of collections, why we're using it, and get feedback on the collection concepts for the metrics test. | January |
Outreach | Meet with stakeholders | January |
Design | Respond to comments and iterate on metrics definitions based on feedback. | February-March |
Evaluate and iterate on metrics design
[edit]Phase | Description | Timeframe |
---|---|---|
Evaluate | Assess metrics and outcomes based on the questions defined in Project evaluation. | February-March 2025 |
Design | Define the v1 set of metrics to implement. Design the implementation of those metrics (which will be a more robust and scaleable implementation than used for the v0 testing). | February-March 2025 |
Implement v1 metrics for MediaWiki core tech docs
[edit]Phase | Description | Timeframe |
---|---|---|
Implement | Implement the metrics as defined in the final v1 proposal, and deploy them. May include: Experiment with implementing new data signals, like technical documentation readability scores, and/or updates to Tech Docs dashboard. | March 2025 |
Project phases
[edit]Research phase
[edit]Measuring the health of technical documentation is not a new problem, but measuring it in the Wikimedia context comes with some additional challenges and nuances. Wikimedia projects, the MediaWiki software, and our doc publishing platforms all have unique qualities that call into question the utility of standard tech docs or wiki assessment techniques, even if we focus just on MediaWiki core software documentation.
Research questions:
Design phase
[edit]Define criteria and requirements, analyzed data, and defined an initial set of metrics to test as part of our iterative design process:
Annual plan context
[edit]Quotation from the Foundation's 2024/2025 annual plan
|
---|
|
Project evaluation
[edit]We will measure the success of this project with qualitative and quantitative methods. In particular, we will be looking at:
- Did we implement a tool and/or process via which data that was not previously used is now used to identify and prioritize documentation work?
- Did we identify and/or provide data about MediaWiki Core's technical documentation quality that was not previously available?
- Are MediaWiki stakeholders better able to assess the quality of a given component of the software thanks to data about tech doc health?
- Are MediaWiki stakeholders better able to assess the developer experience for a given workflow thanks to data about tech doc health?