Continuous integration/Workflow 2013

High level overview
It's important to understand what the tools we have are used for and in which order we assess and process their results.


 * Linting (php-lint, jshint, ..) Static analysis for common mistakes, syntax errors, code quality etc. For example references to APIs not available in old IE, trailing commas that crash IE etc, loose comparison that can lead to side effects etc.
 * Server-side unit testing (PHPUnit) Assert server-side modules works as expected on the unit level.
 * Client-side unit testing (QUnit) Assert client-side modules works as expected on the unit level.
 * ... webkit This catches things that would consistently fail in a browser. I.e. someone breaks a javascript unit test (regardless of whether it is Firefox, Chrome or IE), this catches it. This currently needs to be run manually on.
 * ... browser matrix (TestSwarm, BrowserStack) This distributes the QUnit tests to the browser matrix through TestSwarm via BrowserStack.
 * Integration tests (Selenium) Asserts high-level integration between different components and features that rely on OS interaction (creating an account and logging in, uploading files, drag and drop, editing/saving/reverting/recentchanges etc.). These are written in Selenium and ran semi-automatically via SauceLabs.

Assessment should go from top to bottom. For example we assert that the database is operating properly before we try to request a web page and see if the javascript module for querying the API works properly. If tests show the database doesn't work, there is no point in running any of the API tests.


 * : I'm not sure if there is a good CSS linter out there, but we should have something for it. It would also have to support a way to allow used IE-hack workarounds that use invalid CSS.
 * : JSLint or JSHint (with these settings) should be good. We need to check if there are false positives we need to account for. If there aren't then it should be possible to run JSHint from the command line with NodeJS.
 * more to be added
 * Jenkins-Gerrit (as gerrit user "lintbot") reports a +1 or -1.

At this point people are free to leave comments, commit additional patches, give their opinion via +1 or -1. And eventually someone with the permission to review in this repository will either reject it (-2) or mark it OK (+2).

If it is rejected then the commit story ends until further notice. If it was marked OK, however, then the story continues:

Commit is approved

 * Jenkins-Gerrit plugin gets notified and holds the merge hostage for now. Although the reviewer authorized the merge to master, Jenkins is the one who will actually command Gerrit to perform the merge (or deny it) based on the results of the unit test.


 * "MediaWiki-unittest-gerrit" job starts:
 * reset the clone; update the clone of origin/master; cherry-pick the commit into the job's working directory
 * The child jobs will run one by one now
 * They create their build.
 * -phpunit-* child jobs:
 * Creating a database
 * Setting the appropriate LocalSettings
 * Run the installer
 * Run PHPunit
 * -testswarm child job:
 * Create a snapshot of the working directory and copy it somewhere where it is readable from the web (e.g. http://integration.mediawiki.org/clone/mw/master/1234/1/)
 * Run the installer (SQLite)
 * TestSwarm (see jquery-ui/grunt.js for an example implementation)
 * "addjob" request to TestSwarm with urls for each test suite, pointing to
 * Use testswarm-browserstack to automatically populate the swarm with the needed browsers (through the BrowserShots API).
 * Poll the the TestSwarm API until it is completed (.
 * Garbage collect the temporary snapshot
 * Whenever a child-job is ready it submits the result to the Jenkins database, reports failures to IRC, and a comment to Gerrit as "jenkinsbot" summarizing the test results and a link to the build page.
 * After all child-jobs are ready, either release the hold on the merge and complete it. Or reject it and report an 'X as "jenkinsbot"

The following structure on Jenkins is planned:
 * MediaWiki-unittest-gerrit Includes sub-jobs for running PHPUnit on various database backends and TestSwarm.:
 * universal-linter
 * phpunit-sqlite
 * phpunit-postgres
 * phpunit-databaseless
 * testswarm

*:not(mediawiki/core.git)
For,   etc. anything that doesn't have a dedicated Jenkins project:
 * General-gerrit
 * universal-linter