Browser testing/Running tests
- 1 TL;DR
- 2 Setup
- 3 Browser support
- 4 Getting browser test code
- 5 Getting the necessary software
- 6 Running the tests
- 7 See also
- 8 References
gem install bundler git clone <project> cd <project>/tests/browser bundle install bundle exec cucumber [features/some.feature[:line]]
You may not need to run browser tests at all. If our Continuous Integration reports that an automated test is failing, you can view its code, manually go through its steps in your browser, and if it fails for you, file a bug against the MediaWiki component that has the problem.
However, if you're writing a test or investigating a test, it helps if you can run the browser tests yourself. You don't necessarily need your own local MediaWiki instance, as the tests can load web pages on some MediaWiki host such as test2.wikipedia.org or en.wikipedia.beta.wmflabs.org.
You do need:
- a browser that supports the test framework
- the code of the tests
- the cucumber test program and all its supporting bits and pieces
Recent Firefox and Chrome/Chromium have the WebDriver automated testing support built-in. So too does PhantomJS, the "headless" WebKit browser.
Getting browser test code
For the most part, browser tests reside in the particular MediaWiki extensions being tested, such as VisualEditor and Flow, in a subdirectory named
tests/browser. All these projects are on gerrit.wikimedia.org and are mirrored to Github.
Use git clone to get the relevant repository, for example
$ git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend
$ git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor
(Note that VisualEditor is the only extension with modules, so find the browser tests under /Visualeditor/modules/ve-mw/tests/browser/)
Common code used by all the browser tests in all the extension repositories is maintained in a Ruby gem called mediawiki-selenium gem.
The mediawiki_selenium gem contains shared steps code such as
- visit RandomPage
- reset Preferences
- check for ResourceLoader errors
and manages common aspects of running tests such as
- starting browsers both local and on Sauce Labs
- managing browser capabilities
- running headless tests
- running browser tests in Jenkins
- managing browser behavior according to environment variables such as KEEP_BROWSER_OPEN and REUSE_BROWSER
Getting the necessary software
Ruby software for browser testing
You can either
- follow the instructions in mediawiki-selenium to configure Ruby and install
cucumberand supporting code for your system
- Quality Assurance/Browser testing/Setup instructions has detailed guidance on installing Ruby, git, etc.
- install the MediaWiki-Vagrant virtual machine, and in it enable the
browsertestsrole which will install all the necessary code.
The first takes some work, it's not too bad if you are running Linux or Mac OS X.
The second may be more than you need if you are going to run tests targeting a public MediaWiki instance, since it configures an entire local MediaWiki installation.
A browser must implement the WebDriver interface so that tests can "drive" it. This is built into Firefox (the default browser assumed by our browser tests); for Chrome/chromium you need to install it, e.g. the package
chromium-chromedriver. See the chromedriver page for more information.
To run a regular browser without it drawing windows on your screen, you need to install xvfb so it can draw to an invisible "X virtual framebufffer". In Ubuntu or Debian, something like sudo apt-get install --no-install-recommends xvfb
You can also run tests in the PhantomJS headless WebKit browser. Recent versions have built-in WebDriver support.
Running the tests
For each test run you must tell Cucumber the wiki to visit, supply some additional information if the tests log in, and name the features to test.
- Specify the target wiki that the browser will visit with export MEDIAWIKI_URL=URL/to/target/wiki/ , e.g. http://127.0.0.1:8080/wiki/ for MediaWiki-Vagrant.
- Tests that use the MediaWiki action API need a export MEDIAWIKI_API_URL=URL/to/api.php ,e.g. http://en.wikipedia.beta.wmflabs.org/w/api.php for Betalabs.
- For tests that login:
- create a user named
Selenium_useron the target wiki
- specify the
Selenium_userpassword on the target wiki with export MEDIAWIKI_PASSWORD=Selenium_user_password
- create a user named
- In the
browsertests/tests/browserdirectory, type bundle exec cucumber to run all tests.
- To ignore tests tagged with
@wip(works in progress), run cucumber with the
-wflag. E.g. bundle exec cucumber -w
- To run tests tagged for a particular browser or host computer, run cucumber with the
--tags @some_annotationflag. E.g. bundle exec cucumber --tags @firefox --tags @en.wikipedia.beta.wmflabs.org. Cucumber tags explains these annotations in more detail.
- the tags for each of our Continuous integration browser test jobs are set in its configuration script, click "XXX"
- To run a single test, in the
browsertests/tests/browserdirectory enter bundle exec cucumber features/foo.feature:NN , where
foo.featureis the name of the feature test and NN is any line within the particular scenario you want to test.
- The browser by default is Firefox. You can change the browser by setting the environment variable
BROWSER, other choices include
phantomjs. PhantomJS is a headless WebKit browser.
- The browser specified by the
BROWSERopens windows on your screen (unless it is phantomjs). On Linux with X11 you can run browser tests in a headless mode by setting the environment variable
true. This makes the browser windows appear on an invisible Xvfb virtual framebuffer.
- By default, the browser will close itself at the end of every scenario. You can keep the browser windows around by setting the environment variable
true. (This makes no sense for an invisible run using HEADLESS or phantomjs.)
- You can get screenshots of failures by setting the environment variable
true. This will capture the screen after errors and save to a
export these environment variables, or put all these variables on the command line before invoking bundle exec cucumber, thus
$ HEADLESS=true \ SCREENSHOT_FAILURES=true \ BROWSER=chrome MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ \ MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php \ MEDIAWIKI_USER=Selenium_user \ MEDIAWIKI_PASSWORD=SomePassword \ bundle exec cucumber features/search.feature
Running browser tests at Sauce Labs
Sauce Labs gives you access to a variety of browsers and platforms for testing. For a full list of browsers and OSes supported by Sauce Labs, see their platforms page (the specific Linux variant used by Sauce Labs is Ubuntu LTS). If you don't already have a Sauce Labs account, you can get a free limited account at the Sauce Labs signup page (choose either a Free account or an Open Sauce account).
Before running a browser test via Sauce Labs, you'll need to define five environment variable values:
The first three values are selected from the set of possible values defined on the Sauce Labs platforms page. The last two values are your Sauce Labs user credentials.
For example, to run a browser test via Sauce Labs with Windows XP and Firefox version 27, you would define environment variable values as follows.
export BROWSER=firefox export VERSION=27 export PLATFORM="Windows XP" export SAUCE_ONDEMAND_USERNAME=(username) export SAUCE_ONDEMAND_ACCESS_KEY=(key)
set BROWSER=firefox set VERSION=27 set PLATFORM="Windows XP" set SAUCE_ONDEMAND_USERNAME=(username) set SAUCE_ONDEMAND_ACCESS_KEY=(key)
(key) with your Sauce Labs username and key. You can get the key at https://saucelabs.com/account at the bottom-left part of the screen.
Then if (for example) you want to run the Go to Log in page scenario, you can execute the test with the usual Cucumber command syntax:
bundle exec cucumber features/login.feature:18
Running VisualEditor browser tests via Vagrant
Some of the browser tests have been re-located to the specific code repositories for which they are most relevant. An example is browser tests for Visual Editor, which can be found in the VisualEditor GitHub repo. To run these tests in the MediaWiki-Vagrant virtual environment, a couple of extra steps are necessary in order to make the Cucumber script itself findable. Here are specific steps that should do the trick:
(in the directory where you have cloned the vagrant repo):
$ vagrant enable-role visualeditor $ vagrant up $ vagrant provision $ vagrant ssh -- -X $ cd /srv/VisualEditor/VisualEditor/modules/ve-mw/test/browser $ export MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ $ export MEDIAWIKI_USER=<username> $ export MEDIAWIKI_PASSWORD=<password> $ export PATH=$PATH:/home/vagrant/.gem/ruby/1.9.1/gems/cucumber-1.3.8/bin $ bundle exec cucumber features/<featureName.feature
visualeditor role not only gives you the VE tests, it configures the Vagrant virtual machine to run a local MediaWiki instance with the VisualEditor extension, so you can manually exercise VisualEditor on your local version. To verify the local MediaWiki instance is running successfully, visit http://127.0.0.1:8080/wiki/Main_Page
Running Math extension browser tests via labs-vagrant
Similar to running visual editor browser tests on vagrant, browser tests for the Math extension can be run on labs-vagrant in the following way.
Once you are logged into your instance all you need to do is:
cd /vagrant/mediawiki/extensions/Math/tests/browser/ bundle install HEADLESS=true bundle exec cucumber features/*.feature
Extending timeouts for slow tests
Tests have a default 5 second timeout. This should be adequate for most operations against production or the beta cluster. If you regularly get errors running tests on a slower test wiki, you can locally modify steps.
Any "when_present" clause will take an arg in seconds, e.g.
- Writing browser test code
- Debugging browser tests
- Continuous integration/Browser tests to automatically run browser tests on each commit
- Unfortunately, headless mode does not work on Mac OS, see https://github.com/leonid-shevtsov/headless/issues/31#issuecomment-8933108.
- Matt @ Sauce Labs Help Desk (personal communication to Jeff Hall, December 6, 2013)