Continuous integration/Tutorials/Generating PHP test coverage for a MediaWiki extension

The Wikimedia CI infrastructure supports generating PHP test code coverage for MediaWiki extensions, and publishing reports on https://doc.wikimedia.org/cover-extensions/.

Requirements

Your PHP code is in a folder named includes/ or src/, and in maintenance/ for scripts.
Your PHPUnit tests are in a folder named tests/phpunit/
Your extension installs sucessfully using the SQLite database backend.

Preparing your tests

Your tests will need @covers tags that tell PHPUnit what code is being covered by the tests. These tags can apply to the whole class, or just for a single test function. They can also apply to an entire class or a specific method. See the PHPUnit documentation for more details.

Note that if you are not extending from MediaWikiTestCase, then PHPUnit will not validate the tags until it tries to run and generate coverage. You can use the MediaWikiCoversValidator trait in your test case to ensure the tags are validated as part of normal tests.

Running locally

First you need to edit tests/phpunit/suite.xml in MediaWiki core to adjust the coverage filter:

		<whitelist addUncoveredFilesFromWhitelist="true">
			<directory suffix=".php">../../extensions/FooBar/includes</directory>
			<directory suffix=".php">../../extensions/FooBar/src</directory>
			<directory suffix=".php">../../extensions/FooBar/maintenance</directory>
		</whitelist>

Example diff

diff --git a/tests/phpunit/suite.xml b/tests/phpunit/suite.xml
index e8256ef2cb..44001febf8 100644
--- a/tests/phpunit/suite.xml
+++ b/tests/phpunit/suite.xml
@@ -65,12 +65,9 @@
        </groups>
        <filter>
                <whitelist addUncoveredFilesFromWhitelist="true">
-                       <directory suffix=".php">../../includes</directory>
-                       <directory suffix=".php">../../languages</directory>
-                       <directory suffix=".php">../../maintenance</directory>
-                       <exclude>
-                               <directory suffix=".php">../../languages/messages</directory>
-                       </exclude>
+                       <directory suffix=".php">../../extensions/FooBar/includes</directory>
+                       <directory suffix=".php">../../extensions/FooBar/src</directory>
+                       <directory suffix=".php">../../extensions/FooBar/maintenance</directory>
                </whitelist>
        </filter>
 </phpunit>

Then you can run: php tests/phpunit/phpunit.php --wiki wiki --testsuite extensions --coverage-html coverage extensions/FooBar/tests/phpunit

You can then open ./coverage/index.html in your web browser to view the coverage report. Even if individual tests fail, the coverage report should still get generated.

For xdebug XDEBUG_MODE=coverage should be set.

For the Math extension the following command could be used

mkdir -p ../coverage
XDEBUG_MODE=coverage php -dpcov.enable=1 -dpcov.directory=../coverage -d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit -v -c tests/phpunit/suite.xml ./extensions/Math/tests/phpunit/unit --coverage-html ../coverage  --whitelist $(pwd)/extensions/Math

Running with mediawiki-Docker

See MediaWiki-Docker/Extension/Math#Code coverage report for an example how to run code coverage report in mediawiki-docker

At the moment there is a problem running code coverage for the entire math phpunit directory, but tests can be run individually phab:T378434.

Running in CI

Jenkins can trigger a build of the extension coverage after a commit is merged in the repository. You'll need to send a patch to the integration/config repository. See gerrit:435673 for which jobs need to be added.