Continuous integration/Workflow 2013

I've been thinking and chatting about unit testing at random intervals last few weeks and been thinking about the new Gerrit workflow. Here's a short overview of what I think would be a good plan. I believe most (if not all) of these points were already discussed but I couldn't find a definitive plan so I wrote it up here.


 * Commit comes into Gerrit for mediawiki-core repository.
 * Lint checks run and report to Gerrit. Gerrit-bot user "lintbot" runs immediately and reports a +1 or -1. This bot runs a certain check based on the file-extension. It should be fairly easy to extend or modular so that support for more file types can be added later. For startes it would have:
 * : Simply ` `
 * : I'm not sure if there is a good CSS linter out there, but we should have something for it. It would also have to support a way to allow used IE-hack workarounds that use invalid CSS.
 * : JSLint or JSHint (with these settings) should be good. We need to check if there are false positives we need to account for. If there aren't then it should be possible to run JSHint from the command line with NodeJS.
 * At this point people are free to leave comments, commit additional patches, give their opinion via +1 or -1. And eventually someone with the permission to review in this repository will either reject it (-2) or mark it OK (+2).
 * If it is rejected then the commit story ends until further notice. If it was marked OK, however, then the story continues.
 * Gerrit notifies Jenkins.
 * Our Jenkins plugin gets a revision ID to check. Although the reviewer authorized the merge to master, Jenkins is the one who will actually command Gerrit to perform the merge (or deny it) based on the results of the unit test.
 * Jenkins-plugin creates a new git clone of the mediawiki-core master branch into, and cherry-pick the commit (or in English: "apply the patch").
 * Jenkins-plugin will install MediaWiki from that directory.
 * Jenkins-plugin creates new builds, one for each of the "gerrit" projects:
 * MediaWiki PHPUnit (Gerrit fetcher): Runs phpunit on ./tests/phpunit.
 * Since the PHP unit test is synchronous the process will be able to report back to Jenkins in real-time and mark the build "completed" and "passed" or "failure" right after the test completes.
 * MediaWiki TestSwarm (Gerrit fetcher): Take the QUnit test suites and add a new job for them in TestSwarm. URLs pointing to the MediaWiki install.
 * The QUnit tests are asynchronous because they are executed from a web browser and results are sent back via AJAX. The PHP process adding the job to TestSwarm will return before they are done. The Jenkins-plugin periodically polls the TestSwarm API to aggregate the results and update the Jenkins build.
 * Once the Jenkins-plugin sees that both builds have completed it does 2 things:
 * Schedule the test clone for destruction (clean up of the databases and the MediaWiki source code).
 * Update to Gerrit:
 * It adds a +2 if all are successful builds and -2 if any build had a failure.
 * It also leaves a comment in Gerrit including each builds name, status and url to the build results in Jenkins and TestSwarm.
 * If all builds were successful it gives a green light for the merge and command Gerrit to perform the merge.

Note that there are several factors that make the history of these "(Gerrit fetcher)" projects in Jenkins different than the actual master branch:
 * possibility of the master+cherrypick state being different then the state after the merge (such as differences caused by race conditions)
 * a merge is rejected (in which case it isn't merged)
 * (other differences?)

Therefor there is another set of Jenkins jobs that listen to the actual Git repository branches, not the Gerrit streams. That leaves the following structure on Jenkins:
 * MediaWiki Lint (Gerrit fetcher)
 * MediaWiki PHPUnit (Gerrit fetcher)
 * Includes sub-jobs for different database backends (*-sqlite, *-databaseless)
 * MediaWiki TestSwarm (Gerrit fetcher)
 * MediaWiki PHPUnit (master)
 * Includes sub-jobs for different database backends (*-sqlite, *-databaseless)
 * MediaWiki TestSwarm (master)
 * MediaWiki PHPUnit (REL1_19)
 * MediaWiki TestSwarm (REL1_19)
 * MediaWiki PHPUnit (REL1_18)
 * MediaWiki TestSwarm (REL1_18)

References: