Wikimedia Technical Documentation Team/Documentation tool study

This page contains the final report of the documentation tool study conducted in August-October 2024 by the Wikimedia Foundation Technical Documentation Team (tracked in T371670, research and report by KBach-WMF). The goal of the study was to better understand the tools and processes used by the Wikimedia Movement when creating and maintaining technical documentation . This report contains the main takeaways from the study and recommendations for the next steps.

Methodology

This project consisted of three phases, each with its own assumptions, methods, tools:

Data collection - collecting the information about documentation tools and processes used in repositories hosted on Wikimedia GitLab and Gerrit. This was a manual process that involved investigating repository content, checking the linked resources outside the repository (wiki pages, websites, etc.), and checking the common documentation locations for resources connected to the repository.
Data analysis - loading the data about all repositories and items into a database, and then processing and analyzing it using SQL, Python, and Pandas.
Data report - creating the final presentation and report using Jupyter Lab, and then presenting the study outcomes to the Technical Documentation Team and the general audience.

Repositories

The repositories from Wikimedia GitLab and Gerrit included in the study are as follows.

GitLab:

A large selection of toolforge-repos projects
All affiliate projects
A selection of popular (based on star count) personal and general projects
Small selection of random repositories

Gerrit:

Random selection of non-archived repositories

Items

Each identified documentation tool, process, or product was added to the list of documentation items. For empty repositories and repositories without any documentation, a single item marked Empty repo or No docs was added to the same list.

As a result, an item can represent:

an empty repository
a repository with no documentation
a section on a wiki page, a wiki page, a collection of wiki pages clustered together
a file, a folder with many files, separate files spread throughout a repository
an entire website
a tool used to process or produce any of the former

Data set

The data set contains two data structures in the form of tables, one representing repositories, and one representing items. The total number of repositories included in the survey is 611. The total number of items generated based on these repositories is 952.

In the data set, 80.2% of repositories are not empty and 19.8% are empty. Wikimedia Foundation staff members contributed to 50.74%, volunteers - 48.28%, affiliates - 9%, commercial entities - 2.45%, and others to 0.16% of the repositories.

Most (94%) of the identified 121 empty repositories belonged to tools. Empty repositories were generally excluded from the analysis.

Main takeaways

State of documentation

Over 85% of non-empty repositories have some sort of documentation.

Count	Docs	%
417	With docs	85.1
73	Without docs	14.9

Most popular documentation tools

READMEs and wiki pages are the most common documentation tools with 39.25% and 35.75% repositories respectively using them in documentation. All other tools add up to almost 16%, only seven percentage points above the number of repositories without documentation.

Tool	Count	%
README	314	39.25
Wiki	286	35.75
Other	127	15.88
None	73	9.12

Tools per project type

Looking at which documentation tools are most popular in specific project types:

Wikis are the most popular documentation tool for:

affiliate projects
extensions
projects related to gadgets (tied with READMEs)
projects related to mobile apps
skins
SRE projects (tied with READMEs)

READMEs are the most popular documentation tool for:

personal projects
projects related to analytics
projects related to gadgets (tied with wikis)
RelEng projects
SRE projects (tied with wikis)
tools
other projects

In summary, the top 3 tools per project type are:

In-repo README
- 7 times in 1st spot
- 4 times in 2nd spot
- 0 times in 3rd spot
- 0 times outside top 3
Wiki
- 6 times in 1st spot
- 3 times in 2nd spot
- 2 times in 3rd spot
- 0 times outside top 3
None (that is, no documentation is present)
- 2 times in 2nd spot
- 1 time in 3rd spot
3rd party website
- 2 times in 3rd spot
JSDoc
- 2 times in 3rd spot
doc.wikimedia.org
- 2 times in 3rd spot
In-repo docs
- 1 time in 3rd spot

Tools per documentation type

Wikis are the most popular tool for:

administrator documentation
project docs (though numbers related to project docs aren't fully reliable in this study)
research docs (tied with READMEs)
user docs

READMEs are the most popular tool for:

developer documentation
research docs (tied with READMEs)
other documentation

Documentation tools variety

While READMEs and wikis are the dominant tools used in documentation, there is still considerable variety in documentation tooling, with 35 other tools used in different projects.

Looking at the number of items per documentation tool produces the following complete breakdown. Note that this record set excludes items representing repositories without documentation (None in the previous section).

Tool	Count	%
Wiki	315	41.61
In-repo readme	315	41.61
In-repo docs	26	3.43
3rd-party website	22	2.91
doc.wikimedia.org	15	1.98
Doxygen	8	1.06
JSDoc	6	0.79
In-app docs	5	0.66
In-code documentation	4	0.53
OpenAPI spec	3	0.40
Sphinx	3	0.40
php-code-coverage	3	0.40
Empty repo	3	0.40
Vitepress	2	0.26
HTML docs	2	0.26
Docpub	2	0.26
Istanbul	2	0.26
Custom documentation generator	1	0.13
Unknown tool	1	0.13
MediaWiki API page	1	0.13
Toolforge	1	0.13
PDF	1	0.13
Manpages	1	0.13
Docs in tool	1	0.13
rustdoc	1	0.13
Redoc	1	0.13
Phabricator	1	0.13
LateX	1	0.13
	1	0.13
Docbook	1	0.13
Custom wiki	1	0.13
Jupyter notebooks	1	0.13
Video gif	1	0.13
Demo site	1	0.13
In-repo example	1	0.13
JSDuck	1	0.13
Google Docs	1	0.13
GroovyDoc	1	0.13

Developer documentation in particular uses a large variety of tools.

Documentation type	Tool	Count	% of type total
Developer docs	In-repo readme	141	54.02
Developer docs	Wiki	60	22.99
Developer docs	In-repo docs	14	5.36
Developer docs	Doxygen	8	3.07
Developer docs	JSDoc	6	2.30
Developer docs	3rd-party website	5	1.92
Developer docs	In-code documentation	3	1.15
Developer docs	Sphinx	3	1.15
Developer docs	OpenAPI spec	3	1.15
Developer docs	Vitepress	2	0.77
Developer docs	GroovyDoc	1	0.38
Developer docs	In-repo example	1	0.38
Developer docs	Docpub	1	0.38
Developer docs	In-app docs	1	0.38
Developer docs	Custom documentation generator	1	0.38
Developer docs	Docbook	1	0.38
Developer docs	JSDuck	1	0.38
Developer docs	Docs in tool	1	0.38
Developer docs	MediaWiki API page	1	0.38
Developer docs	Demo site	1	0.38
Developer docs	Custom wiki	1	0.38
Developer docs	PDF	1	0.38
Developer docs	Jupyter notebooks	1	0.38
Developer docs	LateX	1	0.38
Developer docs	Redoc	1	0.38
Developer docs	rustdoc	1	0.38

Least documented projects

Personal projects and tools are the only project types with a significant number of repositories without documentation. 34% and 31% of these projects respectively had no documentation.

Project type/Docs	With docs	Without docs	% without docs
Extension	153.0	3.0	1.92
Gadget	3.0	0.0	0
Other	61.0	2.0	3.17
Other - RelEng	13.0	0.0	0
Other - SRE	4.0	0.0	0
Other - affiliate	22.0	0.0	0
Other - analytics	14.0	3.0	17.65
Other - mobile	2.0	0.0	0
Other - personal	53.0	27.0	33.75
Skin	9.0	0.0	0
Tool	83.0	38.0	31.4

Even in situations where a given repository technically has a README or other documentation, there are instances where that documentation doesn't contain any valuable information. I refer to these items as token documentation.

Token documentation was most common among items related to tool repositories, where 37 out of 161 items (almost 23%) contained token docs.

Instances of token documentation appear more often in READMEs than other tools.

Project type/Tool	In-repo docs	In-repo readme	Wiki
Extension	2.0	9.0	6.0
Gadget	0.0	2.0	0.0
Other	0.0	2.0	0.0
Other - affiliate	0.0	2.0	0.0
Other - analytics	0.0	4.0	0.0
Other - personal	0.0	9.0	0.0
Tool	0.0	37.0	3.0

Documentation presence vs contributor types

While this wasn't the primary purpose of the research, I think it's worth mentioning that based on available data, correlation between contributor types and documentation presence is weak. Specifically, presence of documentation in repositories varies by contributor type involved in the project in two broad groups:

for affiliates and commercial entities, documentation is present in 98% and 93% of repositories respectively
for Wikimedia Foundation staff and volunteers, documentation is present in 88% and 87% of repositories respectively

Documentation absent from repositories by contributor type:

Contributor	Count	Contributor total	% without docs
Affiliates	1	55	1.81
Commercial entities	1	15	6.66
Other	0	1	0
Staff	36	307	11.72
Volunteers	36	287	12.54

Notably, documentation absence is less of a problem in repositories with a higher number of contributor types:

Repository/Contributor type count	1	2	3
With documentation	269.0	122.0	26.0
Without documentation	68.0	3.0	0.0

Recommendations

Based on the results of this study I formulated the following documentation strategy recommendations for the Technical Documentation Team:

Ensure high quality tooling and templates for README files and wiki documentation. This is already available for wiki documentation with style guides and templates at Technical style guides and templates and Toolkit . READMEs could use a bit more attention.
Ensure adequate platform support for less popular documentation tools. Support other Wikimedia Foundation teams in building solutions that work with a broad variety of documentation tools.
Keep monitoring the state of documentation tooling and find ways to facilitate, streamline, and standardize usage of popular tools and processes. This has already been the case since the formation of the Technical Documentation Team as evidenced by the team's ownership of doc.wikimedia.org and the JSDoc migration project.
Seek out opportunities for introducing documentation early in personal repositories and tool projects. This could be done, for example, by improving mechanisms of generating repositories.