Quality Assurance/How to run browser tests


 * First read Understanding browser tests.

You may not need to run browser tests at all. If cloudbees reports that an automated test is failing, you can view its code, manually go through its steps in your browser, and if it fails for you, file a bug against the MediaWiki component that has the problem.

However, if you're writing a test or investigating a test, it helps if you can run the browser tests yourself You don't necessarily need your own local MediaWiki instance, you can run the tests on some MediaWiki host such as test2.wikipedia.org or en.wikipedia.beta.wmflabs.org/ You do need
 * a browser that supports the test framework
 * the code of the tests
 * the cucumber test program and all its supporting bits and pieces

Browser support
Recent Firefox and Chrome/Chromium have the WebDriver automated testing support built-in.

Getting browser test code
Some of the tests are in the MediaWiki browsertests project, https://github.com/wikimedia/qa-browsertests. Others are in particular MediaWiki extensions, such as VisualEditor and Flow, often in a subdirectory named tests/browser. All these projects are on gerrit.wikimedia.org and are mirrored to github.

Use git clone to get the relevant repository, for example

Running the tests

 * clone the repository: git clone ssh://username@gerrit.wikimedia.org:29418/qa/browsertests
 * follow the instructions in README.md to configure Ruby for your system
 * specify the target wiki that the browser will visit with export MEDIAWIKI_URL=URL/to/target/wiki, e.g. http://127.0.0.1:8080/wiki/ for MediaWiki-Vagrant.
 * for tests that login,
 * create a user named Selenium_user on the target wiki
 * specify the Selenium_user password with export MEDIAWIKI_PASSWORD=Selenium_user_password (for some older tests you may need to create a local file config/secret.yml containing mediawiki_password: 'Selenium_user_password'
 * in the browsertests/ directory, type bundle exec cucumber to run all tests (tests that login will fail until you supply the proper passwords)
 * to run a single test, in the browsertests/ directory enter bundle exec cucumber features/foo.feature, where foo.feature" is the name of the feature test to be run.
 * the browser by default is Firefox. You can change the browser by setting the environment variable BROWSER_LABEL
 * By default, the browser will close itself at the end of every scenario. If you want the browser to stay open, set the environment variable  to.

You can put all these variables on one command line, thus

KEEP_BROWSER_OPEN=true MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ MEDIAWIKI_USER=Selenium_user MEDIAWIKI_PASSWORD=SomePassword bundle exec cucumber features/search.feature

Running browser tests at SauceLabs
Sauce Labs gives you access to a variety of browsers and platforms for testing.

Linux/Unix/Mac

Windows Replace (username) and (key) with your Sauce Labs username and key. You can get the key at https://saucelabs.com/account at the bottom-left part of the screen.

Then for example if you want to run the Go to Log in page scenario for example, just run it with this:

bundle exec cucumber features/login.feature:18

Running VisualEditor browser tests via Vagrant
Some of the browser tests have been re-located to the specific code repositories for which they are most relevant. An example is browser tests for Visual Editor, which can be found in the VisualEditor GitHub repo. To run these tests in the MediaWiki-Vagrant virtual environment, a couple of extra steps are necessary in order to make the "cucumber" script itself findable. Here are specific steps that should do the trick:

(in the directory where you have cloned the vagrant repo):

Verify that local MediaWiki instance is running sucessfully via: http://127.0.0.1:8080/wiki/Main_Page