Wikimedia Technical Documentation Team/Doc metrics

Measuring the health of our technical documentation can help us assess the developer experience and the state of the Wikimedia technical ecosystem. This project will identify which metrics are relevant for MediaWiki Core documentation, how we can use those metrics to drive our technical documentation work, and how they can help us identify areas for improvement.

Documentation health metrics for MediaWiki Documentation metrics to help measure key aspects of the developer experience. Start: 2024-07 Team members: Technical Documentation Team Lead: Tricia Burmeister

Project overview

As part of the 24/25 annual plan, the Technical Documentation team's goal is to identify and establish metrics that measure the health of Wikimedia technical documentation, using MediaWiki Core documentation as a test case.

We want to measure doc health because documentation is a core component of the developer experience. This project seeks to explore and define:

• What questions in the field of developer/engineering workflows (see WE6) can documentation metrics help us answer?
• What type of documentation data can help enable informed decision-making about how to improve developers' experience?

One expected outcome of this work is a clearer definition and shared understanding of what it means to drive tech docs work with data in the Wikimedia context, and how we can start to build a measurement approach that we can apply to docs in various Wikimedia technical domains (i.e. beyond MediaWiki core).

Goals

• Define an initial set of documentation health metrics for MediaWiki core technical documentation.
• Implement the metrics for a pilot subset of MediaWiki core docs.

Timeline overview

Timeframe	Phase	Description
July–September 2024	Research	Doc metrics and available data
August–September 2024	Research	Doc collections and content scope
September–October 2024	Design	Draft v1 metrics definitions
September–October 2024	Design	Doc collections for pilot
October 2024	Implement	Doc collections for pilot
November–December 2024	Design & outreach	Community feedback on v1 metrics definitions and doc collections
January 2025	Implement	v1 metrics for pilot doc collections
January-February 2025	Evaluate	Assess the utility of the metrics for the collections we measured, and decide how to proceed or change our approach.

Project milestones

The team plans to complete a research phase to help focus and scope our work, followed by a design and implementation phase where we generate and start using doc health metrics for a pilot collection of technical content.

Define a draft set of v1 metrics definitions for MediaWiki core technical documentation

In progress phab:T372102.

Phase	Description	Timeframe
Research	Define the dimensions of tech docs quality that are relevant for MediaWiki docs. Resolve RQ1.	By August 31
Research	Identify data signals that could help us measure those dimensions. Resolve RQ2.	By August 31
Research & Design	Map available data signals to quality dimensions. Identify signals that may require support from other teams, or extra work to implement.	By September 30
Design	Decide on and define a set of v1 metrics to propose for implementation.	By October 15

Identify doc collections to measure

Phase	Description	Timeframe
Research	Define the scope of what to measure. Resolve RQ3.	By September 30
Design	Use either developer workflows, MediaWiki core code structure, or some other logical framework to identify several content collections to measure.	By October 15
Implement	Use PagePile or other mechanism to create collections.	By October 30

Get feedback and community input on v1 metrics and doc collections

Phase	Description	Timeframe
Outreach	Present v1 metrics proposal for community feedback. Explain the concept of collections, why we're using it, and get feedback on the collection concepts for the metrics pilot.	November
Outreach	Meet with stakeholders	November
Design	Respond to comments and iterate on metrics definitions based on feedback.	November-December

Implement v1 metrics for MediaWiki core tech docs

Phase	Description	Timeframe
Design	Finalize and publish the set of v1 metrics definitions, and the documentation collections we will use to test them.	By December 20
Implement	Implement the metrics as defined in the final v1 proposal, and deploy them for the pilot doc collections. May include: Experiment with implementing new data signals, like technical documentation readability scores, and/or updates to Tech Docs dashboard.	January 2025

Evaluate and plan next steps

Phase	Description	Timeframe
Evaluate	Assess project outcomes based on the questions defined in Project evaluation.	February 2025
Evaluate / Outreach	Get feedback from community and stakeholders about project outcomes and utility of the metrics.	February 2025
Plan	Determine how to iterate on and/or scale metrics based on what we learned. Update documentation for any processes that should now use new documentation metrics.	February-March 2025

Project phases

Research phase

Measuring the health of technical documentation is not a new problem, but measuring it in the Wikimedia context comes with some additional challenges and nuances. Wikimedia projects, the MediaWiki software, and our doc publishing platforms all have unique qualities that call into question the utility of standard tech docs or wiki assessment techniques, even if we focus just on MediaWiki core software documentation.

Research questions

RQ1: Dimensions of doc health

How do we define tech doc health for MediaWiki core?

• Define what technical documentation health means in the Wikimedia context, for the MediaWiki core software.
• Identify where can we use industry-standard definitions of doc health, where can we use wiki-standard definitions of content quality, and where we we may need to diverge from both of those metrics frameworks.

RQ2: Available data

What data do we have, and what do we want?

• Map existing data signals and tools to the quality dimensions we care about, and identify gaps.
• Clarify which Wikimedia analytics tools don't (yet) work, or aren't available, for our technical wikis.

RQ3: Content scope

What pieces of content are meaningful to measure?

• Identify the scope of "MediaWiki Core" documentation."
• Identify collections that can serve as pilot groupings of content for measurement.
• Define content, or characteristics of content, that we could measure but can't really impact...so we can save time and effort by realizing that is out of scope.

Annual plan context

Quotation from the Foundation's 2024/2025 annual plan

Objective: Technical staff and volunteer developers have the tools they need to effectively support the Wikimedia projects. We will continue work started to improve (and scale) development, testing and deployment workflows in Wikimedia Production and expand the definition to include services for tool developers. We also aim to improve our ability to answer frequently asked questions in the field of developer/engineering workflows and audiences and make relevant data accessible to enable informed decision making. Part of this work is to look at practices (or lack of such) that currently present a challenge for our ecosystem. WE 6.1 will "Resolve 5 questions to enable efficiency and informed decisions on developer and engineering workflows and services and make relevant data accessible by the end of Q4." The overarching goal of this work is to reduce the time, workarounds and effort it takes to gain insights in key aspects of the developer experience and enable us to make improvements to engineering and developer workflows and services. —Wikimedia Foundation, Annual Plan/2024-2025. Wiki Experiences: Knowledge platform II (Developer Services). Objective 6.1

Project evaluation

We will measure the success of this project with qualitative and quantitative methods. In particular, we will be looking at:

• Did we implement a tool and/or process via which data that was not previously used is now used to identify and prioritize documentation work?
• Did we identify and/or provide data about MediaWiki Core's technical documentation quality that was not previously available?
• Are MediaWiki stakeholders better able to assess the quality of a given component of the software thanks to data about tech doc health?
• Are MediaWiki stakeholders better able to assess the developer experience for a given workflow thanks to data about tech doc health?