Continuous integration/Tutorials/Generating PHP test coverage for a MediaWiki extension

The Wikimedia CI infrastructure supports generating PHP test code coverage for MediaWiki extensions, and publishing reports on https://doc.wikimedia.org/cover-extensions/.

Requirements

Your PHP code is in a folder named includes/ or src/, and in maintenance/ for scripts.
Your PHPUnit tests are in a folder named tests/phpunit/
Your extension installs sucessfully using the SQLite database backend.

Preparing your tests

Your tests will need @covers tags that tell PHPUnit what code is being covered by the tests. These tags can apply to the whole class, or just for a single test function. They can also apply to an entire class or a specific method. See the PHPUnit documentation for more details.

Note that if you are not extending from MediaWikiTestCase, then PHPUnit will not validate the tags until it tries to run and generate coverage. You can use the MediaWikiCoversValidator trait in your test case to ensure the tags are validated as part of normal tests.

Running locally

First, you need to create the PHPUnit configuration. Therefore, you can use the following composer script (**Math** is the example for an extension): composer phpunit:coverage-edit -- extensions/Math

Then you can run: XDEBUG_MODE=coverage composer phpunit -- --testsuite extensions --coverage-html coverage extensions/Math/tests/phpunit --exclude-group Stub

Example diff

> phpunit '--testsuite' 'extensions' '--coverage-html' 'coverage' 'extensions/Math/tests/phpunit' '--exclude-group' 'Stub'
Using PHP 8.2.27
Running with MediaWiki settings because there might be integration tests
PHPUnit 9.6.21 by Sebastian Bergmann and contributors.

.............................................................   61 / 2192 (  2%)
.............................................................  122 / 2192 (  5%)
..........SSSSSS.............................................  183 / 2192 (  8%)
.............................................................  244 / 2192 ( 11%)
.............................................................  305 / 2192 ( 13%)
.............................................................  366 / 2192 ( 16%)
.............................................................  427 / 2192 ( 19%)
.............................................................  488 / 2192 ( 22%)
.............................................................  549 / 2192 ( 25%)
.............................................................  610 / 2192 ( 27%)
.............................................................  671 / 2192 ( 30%)
.............................................................  732 / 2192 ( 33%)
.............................................................  793 / 2192 ( 36%)
.............................................................  854 / 2192 ( 38%)
.............................................................  915 / 2192 ( 41%)
.............................................................  976 / 2192 ( 44%)
............................................................. 1037 / 2192 ( 47%)
............................................................. 1098 / 2192 ( 50%)
............................................................. 1159 / 2192 ( 52%)
............................................................. 1220 / 2192 ( 55%)
............................................................. 1281 / 2192 ( 58%)
............................................................. 1342 / 2192 ( 61%)
............................................................. 1403 / 2192 ( 64%)
............................................................. 1464 / 2192 ( 66%)
............................................................. 1525 / 2192 ( 69%)
............................................................. 1586 / 2192 ( 72%)
............................................................. 1647 / 2192 ( 75%)
............................................................. 1708 / 2192 ( 77%)
............................................................. 1769 / 2192 ( 80%)
............................................................. 1830 / 2192 ( 83%)
............................................................. 1891 / 2192 ( 86%)
............................................................. 1952 / 2192 ( 89%)
............................................................. 2013 / 2192 ( 91%)
............................................................. 2074 / 2192 ( 94%)
............................................................. 2135 / 2192 ( 97%)
.........................................................     2192 / 2192 (100%)

Time: 00:19.597, Memory: 353.50 MB

OK, but incomplete, skipped, or risky tests!
Tests: 2192, Assertions: 4139, Skipped: 6.

Generating code coverage report in HTML format ... done [00:00.908]


You should really speed up these slow tests (>100ms)...
 1. 1057ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Big"
 2. 229ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Literals (1)"
 3. 204ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "FUN_AR1"
 4. 149ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Literals (3)"
 5. 133ms to run MediaWiki\\Extension\\Math\\Tests\\PreferencesIntegrationTest::testMathOptionRegistered
 6. 123ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Matrices"

You can then open ./coverage/index.html in your web browser to view the coverage report. Even if individual tests fail, the coverage report should still be generated.

Running with mediawiki-Docker

See MediaWiki-Docker/Extension/Math#Code coverage report for an example of how to run code coverage report in mediawiki-docker

Running in CI

Jenkins can trigger a build of the extension coverage after a commit is merged in the repository. You'll need to send a patch to the integration/config repository. See gerrit:435673 for which jobs need to be added.