This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on any information on this page.
See Selenium instead.
Automated testing of MediaWiki within various browsers to improve quality
Why automate tests at the browser level? They find bugs!
Browser tests are most useful for testing:
- Features requiring navigation: unit tests don't usually navigate.
- Fragile Features: if something breaks often, we want to test it regularly.
- Critical Features: features that must work all the time.
- Beta features: checking that they work in all browsers.
It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.
- We run periodic browser tests via Jenkins on SauceLabs to test the latest master code on the beta cluster in multiple browsers
- Some projects run browser tests on each check-in to gerrit as part of Continuous integration, see Continuous integration/Browser tests.
We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.
How to contribute
Investigate a broken issue
- Analyze failing tests in Jenkins browser test runs.
- Investigate a recent test that's failing
- Drop in to #wikimedia-releng connect and ask if it's a known issue.
- Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
- Watch the screencast of the failing test and/or view the test's code.
- If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.
This doesn't require any local code at all, it's all on the web.
Fixing a broken issue
The next level is to fix broken tests and write new ones.
- Follow the browser testing setup instructions
- Run browser tests yourself, pointing to some MediaWiki install
- If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
- If you find an error in a test, file a bug.
- Going further, contribute a fix for it in gerrit.
- You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.
- Current test results.
- We use given-when-then testing format.
- Cross-browser testing is a resource often use for specific testing