Continuous integration/Workflow 2013

High-level overview
It's important to understand what the tools we have are used for and in which order we assess and process their results.


 * Linting (php-lint, jshint, ..) Static analysis for common mistakes, syntax errors, code quality etc. For example references to APIs not available in old IE, trailing commas that crash IE etc, loose comparison that can lead to side effects etc.
 * : JSHint (with these settings).
 * :  (should use [//svn.wikimedia.org/viewvc/mediawiki/trunk/tools/code-utils/lint.php?view=markup&pathrev=85372 lint.php] instead for performance), we also run PHP CodeSniffer on some projects.
 * : I'm not sure if there is a good CSS linter out there, but we should have something for it. It would also have to support a way to allow used IE-hack workarounds that use invalid CSS.
 * Server-side unit testing (PHPUnit) Assert server-side modules works as expected on the unit level.
 * Client-side unit testing (QUnit) Assert client-side modules works as expected on the unit level.
 * ... webkit This catches things that would consistently fail in a browser. I.e. someone breaks a JavaScript unit test (regardless of whether it is Firefox, Chrome or IE), this catches it.
 * ... browser matrix This will distribute the QUnit tests to the browser matrix (through TestSwarm and BrowserStack, or with Grunt and Saucelabs).
 * Integration tests (Selenium) Asserts high-level integration between different components and features that rely on OS interaction (creating an account and logging in, uploading files, drag and drop, editing/saving/reverting/recentchanges etc.). Our browser tests run regularly at https://wmf.ci.cloudbees.com/

Assessment should go from top to bottom. For example we assert that the database is operating properly before we try to request a web page and see if the JavaScript module for querying the API works properly. If tests show the database doesn't work, there is no point in running any of the API tests.

on each changed file that is php-like
 * phpunit: Various different groups of phpunit executed (databaseless and on sqlite)
 * qunit: Running javascript unit tests in PhantomJS
 * jenkins-bot leaves review on Gerrit.