Why automate tests at the browser level? They find bugs!
Browser tests are most useful for testing:
- Features requiring navigation: unit tests don't usually navigate.
- Fragile Features: if something breaks often, we want to test it regularly.
- Critical Features: features that must work all the time.
- Beta features: checking that they work in all browsers.
It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.
- We run periodic browser tests via Jenkins on SauceLabs to test the latest master code on the beta cluster in multiple browsers
- Some projects run browser tests on each check-in to gerrit as part of Continuous integration, see Continuous integration/Browser tests.
We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.
How to contribute
Investigate a broken issue
- Analyze failing tests in Jenkins browser test runs.
- Investigate a recent test that's failing
- Drop in to and ask if it's a known issue.
- Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
- Watch the screencast of the failing test and/or view the test's code.
- If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.
This doesn't require any local code at all, it's all on the web.
Fixing a broken issue
The next level is to fix broken tests and write new ones.
- Follow the browser testing setup instructions
- Run browser tests yourself, pointing to some MediaWiki install
- If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
- If you find an error in a test, file a bug.
- Going further, contribute a fix for it in gerrit.
- You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
- Check the list of EASY tasks – especially good for new contributors.
None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.