Quality Assurance/Browser testing

From MediaWiki.org
Jump to: navigation, search
Noun project 8974.svg

Status[edit | edit source]


After two years of using a third-party host to run browser test builds in Jenkins, this month we have completed the migration of those builds to Jenkins hosted by the Wikimedia Foundation. Hosting our browser test builds ourselves gives us more control over every aspect of running the browser tests, as well as the potential to run them faster than previously possible. Particular thanks to Antoine Musso, whose work made it possible. Simultaneously, we have also ported all of the remaining tests from the /qa/browsertest repository either to /mediawiki/core or to their relevant extension. This gives us the ability to package browser-based acceptance tests with the release of MediaWiki itself. After more than two years evolving the browser testing framework across WMF, the /qa/browsertests repository is retired, and all if its functions now reside in the repositories of the features being tested.

Rationale[edit | edit source]

Why automate tests at the browser level? They find bugs!

Browser tests are most useful for:

  • Beta features: checking that they work in all browsers.
  • JavaScript: different browsers interpret JS in different ways.
  • Features requiring navigation: unit tests don't navigate.
  • Fragile: if something breaks often, we want to test it regularly.
  • Critical: features that must work all the time.

It's tedious to test manually Wikimedia sites within the multiple browser/operating system combinations that are out there, so we do automated testing to find errors and assure quality.

We use Cucumber because it lets you write tests in plain English; you don't need to know Ruby. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.

Check some real examples. .

How to contribute[edit | edit source]

Here's one way to dive in

  • Analyze test results and reporting failures discovered by way of Jenkins on cloudbees.
  • Investigate a test that's failing
    • drop in to #wikimedia-qaconnect and ask if it's a known issue
    • if you have a cloudbees account, navigate to the Test Result and edit its description with what you figured out, e.g. "Logged in Selenium user made post but it shows up as anonymous contribution (?!) -- spage"
  • Watch the screencast of the failing test and/or view the test's code.
  • If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test.

This doesn't require any local code at all, it's all on the web.

The next level is to fix broken tests and write new ones.

  • Get the test code on your machine.
  • Run browser tests yourself, pointing to some MediaWiki install
  • If the test fails for you but the feature under test seems to be working, try fixing the test, and rerun it.
    • If you find an error in a test, file a bug...
    • and better yet contribute a fix for it in gerrit.
  • You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
  • Check the list of EASY tasks – especially good for new contributors.

None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installing some of the tools you need and checks out the MediaWiki source code. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.

If you are interested in automated browser tests, join the proposed MediaWiki Group Browser testing.

Resources[edit | edit source]