Continuous integration/Understanding build failures


 * This is adapted from a post by Ashar Voultoiz on wikitech-l and archived on gmane

Max Seminik raised the issue of codurr spamming the #mediawiki channel with a useless list of revisions that broke the tests.

Example:

[20:07:03] Something broke. See . Possible culprits: tstarling/r77011 ialex/r77012 /r77014 mglaser/r77021 /r77022 maxsem/r77030 ashley/r77033 tparscal/r77042 /r77052 /r77053 /r77054 /r77055 Raymond_/r77056 platonides/r77058 krinkle/r77063

Ashar repaired cruise control a few days after the Berlin Hackathon 2011. At that points tests were broken and Ashar fixed most of them. Then Ashar added back the Database and Parser groups which added some more interesting tests breakage (including the order in which files are loaded by PHPUnit).

Since cruisecontrol *does not remember the states of tests* from previous builds, it just assume the current build broke everything and hence report all the old breakages on each build :-(

To fix them you have to get to our CruiseControl webpage at http://ci.tesla.usability.wikimedia.org/cruisecontrol/ Clicking on 'mw' in the orange box will show you results for the latest build. On that page, you will find the latest revision tested.

At the moment the original post was wrote, we could see : #288 - testParserTest with data set #287 

Clicking on it will show the long list of tests. Look for the above message in the list, next to it is a "Failure >>" link which show the test output. In this case:

ParserTests::testParserTest with data set #287 ('pre-save transform: Signature expansion in nowiki tags (bug 93)', 'Shall not expand: 

You want to reproduce it locally: $ cd tests $ php parserTests.php --quiet --filter 'bug 93' This is MediaWiki version 1.19alpha (r89273). Passed 0 of 1 tests (0%)... 1 tests failed! $ echo $? # show PHPUnit exit code: 1 $

We have to find out the revision that broken the test, trying svn blame: $ svn blame tests/parser/parserTests.txt | grep 'bug 93' 89191 platonides pre-save transform: Signature expansion in nowiki tags (bug 93) $

So probably introduced by r89191 .. Check it out and run tests :

$ svn co -r 89191 $ php parserTests.php --quiet --filter 'bug 93' This is MediaWiki version 1.19alpha (r89191). Passed 0 of 1 tests (0%)... 1 tests failed! $

That one was broken on commit anyway :

The BlockTest are broken too, most probably due to its rewrite. You can easily reproduce the test suite and most project manager we love the easy report functionality (--testdox):

$ cd tests/phpunit $ ./phpunit.php --filter BlockTest --testdox PHPUnit 3.5.13 by Sebastian Bergmann.

ApiBlock [ ] Make normal block

Block [ ] Initializer functions return correct block [x] Bug 26425 block timestamp defaults to time [ ] Bug 29116 load with empty ip   [ ] Bug 29116 new from target with empty ip  $

Have fun :-)