Continuous integration/Browser tests

Now that we have a reliable job for running MW-Selenium browser tests, you can have Jenkins execute a subset of your tests for every patchset submitted to Gerrit.

What the job does
Reusing much of the JJB configuration originally created for running QUnit tests, it:


 * 1) Checks out master branches of core and the extension dependencies along with the current Gerrit patchset
 * 2) Creates a fresh database and installation of MediaWiki to run on localhost
 * 3) Sets all the necessary MW-Selenium environment variables to point to the local MW installation
 * 4) Executes all scenarios in the test suite tagged with
 * 5) MW-Selenium will start a local headless Xvfb display and execute the tests using the local browser you've configured in
 * 6) Cleans up

Requirements
Your test suite must be using MW-Selenium 1.4 or greater, as it contains a few important fixes to Cucumber logging and browser startup, and a user factory system that creates new accounts for each scenario as they're needed.

Check your version of MW-Selenium by running  in the project directory. If you're already using a 1.x version, the upgrade should be trivial. Just increment the version in  and run   (or   if you run into dependency issues).

If you're running an older version like 0.4.1, the upgrade process will be a little more involved. Start by following the upgrade documentation, and reach out to if you need help.

Setting up your test suite to run in CI
Once you're using MW-Selenium 1.4 or greater, add an  section to the   file in your tests directory that will tell the CI builder how to run the tests. It really just needs to enable the user factory, and specify a browser ( or  ).

Next, simply tag some scenarios with. Choose scenarios that you know are fairly stable but will also give you broad coverage.

Configuring JJB for your extension
Schedule the tests to run upon each commit to Gerrit by modifying the JJB configuration for your extension. You might want to start by adding or modifying the  pipeline, which you can manually trigger to run by adding a review comment "check experimental" to a patch in Gerrit. After you've vetted your set of tests for stability in this environment, move the job to the  and   pipelines. Again, hit up if you need help.

Limitations

 * No video recording—but we're working on it!
 * Jobs are run with local browser for performance reasons, so only Chrome 43 and Firefox 39 are available
 * The fresh MediaWiki installation only sets up one wiki, so you can't test interwiki features