Selenium/Ruby/Running tests

You may not need to run browser tests at all. If Cloudbees reports that an automated test is failing, you can view its code, manually go through its steps in your browser, and if it fails for you, file a bug against the MediaWiki component that has the problem.

However, if you're writing a test or investigating a test, it helps if you can run the browser tests yourself. You don't necessarily need your own local MediaWiki instance, as the tests can load web pages on some MediaWiki host such as test2.wikipedia.org or en.wikipedia.beta.wmflabs.org.

You do need:
 * a browser that supports the test framework
 * the code of the tests
 * the cucumber test program and all its supporting bits and pieces

Browser support
Recent Firefox and Chrome/Chromium have the WebDriver automated testing support built-in. So too does PhantomJS, the "headless" WebKit browser.

Getting browser test code
Some of the tests are in the MediaWiki browsertests project. Others are in particular MediaWiki extensions, such as VisualEditor and Flow, often in a subdirectory named tests/browser. All these projects are on gerrit.wikimedia.org and are mirrored to Github.

Use git clone to get the relevant repository, for example

Getting the necessary software
You can either or
 * follow the instructions in README.md to configure Ruby and install  and supporting code for your system
 * Quality Assurance/Browser testing/Setup instructions has detailed guidance on installing Ruby, git, etc.
 * install the MediaWiki-Vagrant virtual machine, and in it enable the  role which will install all the necessary code.

The first takes some work, it's not too bad if you are running Linux or Mac OS X.

The second is more than you need if you are going to run tests on a public MediaWiki instance, since it configures an entire local MediaWiki installation.

Running the tests
For each test run you must tell Cucumber where to run the tests, and supply some additional information if the tests log in.


 * Specify the target wiki that the browser will visit with export MEDIAWIKI_URL=URL/to/target/wiki, e.g. http://127.0.0.1:8080/wiki/ for MediaWiki-Vagrant.
 * For tests that login
 * create a user named  on the target wiki
 * specify the  password on the target wiki with export MEDIAWIKI_PASSWORD=Selenium_user_password
 * the Selenium_user passwords on some of our test environments where we run Continuous integration tests are stored in Jenkins configuration.
 * In the  directory, type bundle exec cucumber to run all tests.


 * To run a single test, in the  directory enter bundle exec cucumber features/foo.feature, where   is the name of the feature test to be run.
 * The browser by default is Firefox. You can change the browser by setting the environment variable, other choices include chrome and phantomjs.
 * You can run browser tests in a headless mode, meaning that the browser specified by the  value will run in the background via Xvfb . To run tests in this mode, export the   environment variable with a value of.
 * By default, the browser will close itself at the end of every scenario. If you want the browser to stay open, set the environment variable  to.

You can  these environment variables, or put all these variables on the command line before invoking bundle exec cucumber, thus $ KEEP_BROWSER_OPEN=true \ HEADLESS=true \ MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ \ MEDIAWIKI_USER=Selenium_user \ MEDIAWIKI_PASSWORD=SomePassword \ bundle exec cucumber features/search.feature

Running browser tests at SauceLabs
Sauce Labs gives you access to a variety of browsers and platforms for testing. For a full list of browsers and OSes supported by SauceLabs, see their platforms page (the specific Linux variant used by SauceLabs is Ubuntu LTS ).

Linux/Unix/Mac:

Windows: Replace  and   with your Sauce Labs username and key. You can get the key at https://saucelabs.com/account at the bottom-left part of the screen.

Then for example if you want to run the Go to Log in page scenario for example, just run it with this:

bundle exec cucumber features/login.feature:18

Running VisualEditor browser tests via Vagrant
Some of the browser tests have been re-located to the specific code repositories for which they are most relevant. An example is browser tests for Visual Editor, which can be found in the VisualEditor GitHub repo. To run these tests in the MediaWiki-Vagrant virtual environment, a couple of extra steps are necessary in order to make the Cucumber script itself findable. Here are specific steps that should do the trick:

(in the directory where you have cloned the vagrant repo):

Enabling the  role not only gives you the VE tests, it configures the Vagrant virtual machine to run a local MediaWiki instance with the VisualEditor extension, so you can manually exercise VisualEditor on your local version. To verify the local MediaWiki instance is running successfully, visit http://127.0.0.1:8080/wiki/Main_Page