Quality Assurance/Browser testing

From MediaWiki.org
Jump to: navigation, search
Noun project 8974.svg

Rationale[edit | edit source]

Why automate tests at the browser level? They find bugs!

Browser tests are most useful for testing:

  • JavaScript: different browsers interpret JS in different ways.
  • Features requiring navigation: unit tests don't usually navigate.
  • Fragile Features: if something breaks often, we want to test it regularly.
  • Critical Features: features that must work all the time.
  • Beta features: checking that they work in all browsers.

It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.

We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.

How to contribute[edit | edit source]

Investigate a broken issue[edit | edit source]

  • Analyze failing tests in Jenkins browser test runs.
  • Investigate a recent test that's failing
    • Drop in to #wikimedia-relengconnect and ask if it's a known issue.
    • Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
    • Watch the screencast of the failing test and/or view the test's code.
  • If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.

This doesn't require any local code at all, it's all on the web.

Fixing a broken issue[edit | edit source]

The next level is to fix broken tests and write new ones.

  • Follow the browser testing setup instructions
  • Run browser tests yourself, pointing to some MediaWiki install
  • If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
    • If you find an error in a test, file a bug.
    • Going further, contribute a fix for it in gerrit.
  • You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
  • Check the list of EASY tasks – especially good for new contributors.

None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.

Resources[edit | edit source]

Subpages[edit | edit source]

Debugging Environment abstraction layer Examples
Guidelines and good practices Meetings Meetings/2015-06-02
Meetings/2015-06-09 Meetings/2015-06-16 Meetings/2015-06-23
Meetings/2015-07-07 Meetings/2015-08-25 Meetings/Notes
Monitoring results Refactoring 2014 Running tests
Search features Setup instructions Shared features
Test backlog Very Basic Howto Writing feature descriptions
Writing tests status