Continuous integration/Workflow 2013

I've been thinking and chatting about unit testing at random intervals last few weeks and been thinking about the new Gerrit workflow. Here's a short overview of what I think would be a good plan. I believe most (if not all) of these points were already discussed but I couldn't find a definitive plan so I wrote it up here.


 * Commit comes into Gerrit for mediawiki-core repository.
 * Lint checks run and report to Gerrit.
 * Gerrit-bot user "lint-php" runs immediately and reports a +1 or -1.
 * This would simply be a 'php -l' of all *.php files. I believe such bot already exists.
 * Gerrit-bot user "lint-css" runs immediately and reports a +1 or -1.
 * I'm not sure if there is a good CSS linter out there, but we should have something for it. However it would have to allow passing of known IE-hack workarounds that use invalid CSS.
 * Gerrit-bot user "lint-js" runs immediately and reports a +1 or -1.
 * JSLint or JSHint with custom settings should be good, but I'm not sure if there are false positives we need to account for. If there aren't then it should be possible to run JSHint from the command line with NodeJS.
 * At this point people are free to leave comments, commit additional patches, give their opinion via +1 or -1. And eventually someone with the permission to review in this repository will either reject it (-2) or mark it OK (+2).
 * If it is rejected then the commit story ends until further notice. If it was marked OK, however, then the story continues.
 * Gerrit notifies Jenkins.
 * Our Jenkins plugin gets a revision ID to check. Although the reviewer authorized the merge to master, Jenkins is the one who will actually command Gerrit to perform the merge (or deny it) based on the results of the unit test.
 * Jenkins-plugin creates a new git clone of the mediawiki-core master branch into, and cherry-pick the commit (or in English: "apply the patch").
 * Jenkins-plugin will install MediaWiki from that directory.
 * Jenkins-plugin creates new builds, one for each of the "gerrit" projects :
 * MediaWiki Gerrit PHP: Runs phpunit on ./tests/phpunit.
 * Since the PHP unit test is synchronous the process will be able to report back to Jenkins in real-time and mark the build "completed" and "passed" or "failure" right after the test completes.
 * MediaWiki Gerrit JS: Take the QUnit test suites and add a new job for them in TestSwarm. URLs pointing to the MediaWiki install.
 * The QUnit tests are asynchronous because they are executed from a web browser and results are sent back via AJAX. The PHP process adding the job to TestSwarm will return before they are done. The Jenkins-plugin periodically polls the TestSwarm API to aggregate the results and update the Jenkins build.
 * Once the Jenkins-plugin sees that both builds have completed it does 2 things:
 * Schedule the test clone for destruction (clean up of the databases and the MediaWiki source code).
 * Update to Gerrit:
 * It adds a +2 if all are successful builds and -2 if any build had a failure.
 * It also leaves a comment in Gerrit including each builds name, status and url to the build results in Jenkins and TestSwarm.
 * If all builds were successful it gives a green light for the merge and command Gerrit to perform the merge.

Note that due to the possibility of the master+cherrypick state being different then the state after the merge (such as differences caused by race conditions), there is another set of Jenkins projects listening to the actual master, which will also do all these builds. However contrary to the builds triggered by a merge-approval, these builds are not tied to one exact revision. These builds will instead report their results to IRC (not to Gerrit). And if the new master contains a failure, naturally no other merges can take place because even if someone approves a revision at this point, unless the revision being cherry-picked onto the master fixes this regression, Jenkins won't give a green light for the merge.

References: