Selenium/Ruby/Browser testing

Rationale
Why automate tests at the browser level? They find bugs!

Browser tests are most useful for testing:
 * JavaScript: different browsers interpret JS in different ways.
 * Features requiring navigation: unit tests don't usually navigate.
 * Fragile Features: if something breaks often, we want to test it regularly.
 * Critical Features: features that must work all the time.
 * Beta features: checking that they work in all browsers.

It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.

We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.

Investigate a broken issue
This doesn't require any local code at all, it's all on the web.
 * Analyze failing tests in Jenkins browser test runs.
 * Investigate a recent test that's failing
 * Drop in to and ask if it's a known issue.
 * Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
 * Watch the screencast of the failing test and/or view the test's code.
 * If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.

Fixing a broken issue
The next level is to fix broken tests and write new ones.
 * Follow the browser testing setup instructions
 * Run browser tests yourself, pointing to some MediaWiki install
 * If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
 * If you find an error in a test, file a bug.
 * Going further, contribute a fix for it in gerrit.
 * You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
 * Check the list of EASY tasks – especially good for new contributors.

None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.

Resources

 * Current test results.
 * We use given-when-then testing format.