Jump to content

Wikimedia Technical Documentation Team/Doc metrics

From mediawiki.org

Measuring the health of our technical documentation can help us assess the developer experience and the state of the Wikimedia technical ecosystem. This project will identify which metrics are relevant for MediaWiki documentation, how we can use those metrics to drive our technical documentation work, and how they can help us identify areas for improvement.

Project overview

[edit]

As part of the 24/25 annual plan, the Technical Documentation team's goal is to identify and establish metrics that measure the health of Wikimedia technical documentation, using MediaWiki Core documentation as a test case.

We want to measure doc health because documentation is a core component of the developer experience. This project seeks to explore and define:

  • What questions in the field of developer/engineering workflows (see WE6) can documentation metrics help us answer?
  • What type of documentation data can help enable informed decision-making about how to improve developers' experience?

One expected outcome of this work is a clearer definition and shared understanding of what it means to drive tech docs work with data in the Wikimedia context, and how we can start to build a measurement approach that we can apply to docs in various Wikimedia technical domains (i.e. beyond MediaWiki core).

Goals

[edit]
  • Define an initial set of documentation health metrics for MediaWiki core technical documentation.
  • Implement the metrics for a pilot subset of MediaWiki core docs.

Timeline overview

[edit]
Timeframe Phase Description
July– Research Doc metrics and available data
August–September 2024 Design Draft v0 metrics definitions
September– Research/design Doc collections and content scope
September– Implement Doc collections for test
Test Measurement of doc collections for test of v0 metrics
November - December 2024 Evaluate Measurement of doc collections for test of v0 metrics
Decide Determine whether to proceed with gathering community feedback and metrics implementation

Project milestones

[edit]

The team plans to complete a research phase to help focus and scope our work, followed by a design and implementation phase where we generate and start using doc health metrics for a pilot collection of technical content.

Define a draft set of v0 metrics definitions for MediaWiki core technical documentation

[edit]

Yes Done phab:T372102.

Phase Description Timeframe
Research Defined the dimensions of tech docs quality that are relevant for MediaWiki docs. Resolved RQ1. By August 31
Research Identified data signals that could help us measure those dimensions. Resolved RQ2. By August 31
Research & Design Mapped available data signals to quality dimensions. Identified signals that may require support from other teams, or extra work to implement. By September 15
Design Decided on and defined a set of v0 proposed metrics to explore with sample doc collections. By September 15

Identify doc collections to measure

[edit]

Yes Done phab:T374722.

Phase Description Timeframe
Research Define the scope of what to measure. Resolve RQ3. By October 15
Design Use either developer workflows, MediaWiki core code structure, or some other logical framework to identify several content collections to measure. By October 30
Implement Use PagePile or other mechanism to create collections. By October 30

Test v0 metrics on doc collections

[edit]

In progress In progress phab:T379431

Phase Description Timeframe
Implement/Test Test measurement of doc collections using v0 metrics (likely manual assessment). By Nov 15
Evaluate Use of v0 metrics on sample doc collections. By Dec 30

Get feedback and community input on v0 metrics and doc collections

[edit]
Phase Description Timeframe
Outreach Present v0 metrics proposal for community feedback. Explain the concept of collections, why we're using it, and get feedback on the collection concepts for the metrics test. January
Outreach Meet with stakeholders January
Design Respond to comments and iterate on metrics definitions based on feedback. February-March

Evaluate and iterate on metrics design

[edit]
Phase Description Timeframe
Evaluate Assess metrics and outcomes based on the questions defined in Project evaluation. February-March 2025
Design Define the v1 set of metrics to implement. Design the implementation of those metrics (which will be a more robust and scaleable implementation than used for the v0 testing). February-March 2025

Implement v1 metrics for MediaWiki core tech docs

[edit]
Phase Description Timeframe
Implement Implement the metrics as defined in the final v1 proposal, and deploy them. May include: Experiment with implementing new data signals, like technical documentation readability scores, and/or updates to Tech Docs dashboard. March 2025

Project phases

[edit]

Research phase

[edit]

Measuring the health of technical documentation is not a new problem, but measuring it in the Wikimedia context comes with some additional challenges and nuances. Wikimedia projects, the MediaWiki software, and our doc publishing platforms all have unique qualities that call into question the utility of standard tech docs or wiki assessment techniques, even if we focus just on MediaWiki core software documentation.

Research questions:

Design phase

[edit]

Define criteria and requirements, analyzed data, and defined an initial set of metrics to test as part of our iterative design process:

Annual plan context

[edit]
Quotation from the Foundation's 2024/2025 annual plan 

Objective: Technical staff and volunteer developers have the tools they need to effectively support the Wikimedia projects. We will continue work started to improve (and scale) development, testing and deployment workflows in Wikimedia Production and expand the definition to include services for tool developers. We also aim to improve our ability to answer frequently asked questions in the field of developer/engineering workflows and audiences and make relevant data accessible to enable informed decision making. Part of this work is to look at practices (or lack of such) that currently present a challenge for our ecosystem.

WE 6.1 will "Resolve 5 questions to enable efficiency and informed decisions on developer and engineering workflows and services and make relevant data accessible by the end of Q4." The overarching goal of this work is to reduce the time, workarounds and effort it takes to gain insights in key aspects of the developer experience and enable us to make improvements to engineering and developer workflows and services.


Project evaluation

[edit]

We will measure the success of this project with qualitative and quantitative methods. In particular, we will be looking at:

  • Did we implement a tool and/or process via which data that was not previously used is now used to identify and prioritize documentation work?
  • Did we identify and/or provide data about MediaWiki Core's technical documentation quality that was not previously available?
  • Are MediaWiki stakeholders better able to assess the quality of a given component of the software thanks to data about tech doc health?
  • Are MediaWiki stakeholders better able to assess the developer experience for a given workflow thanks to data about tech doc health?