Selenium/Ruby/Writing tests

How To Run and Write WMF Browser Tests
There are three main aspects to manage for automated browser tests in this system: Features and Scenarios; Page Objects; and test steps. Also, there are certain aspects of the test environment of which any user should be aware.

Potential contributors who are not programmers will be most interested in the description of Features and Scenarios. Those who want to dig a bit deeper into how this project works will be interested in Page Objects. People actually creating end-to-end tests will be particularly interested in information about test steps. Finally, anyone wanting to run tests locally needs to aware of certain requirements for the local test environment.

Instructions for running the tests locally are at the end.

All code is at Gerrit (qa/browsertests) and more readable mirror is at GitHub (wikimedia/qa-browsertests).

Features and Scenarios
Features and Scenarios in Cucumber are an implementation of a design pattern for Acceptance Test Driven Development, ATDD. They follow a generally accepted convention for such tests: Feature: My Feature Scenario: Testing some aspect of My Feature Given When  Then  Any Given/When/Then statement may also have an arbitrary number of "And" clauses: Scenario: Testing some complicated aspect of My Feature Given And When  And  And  Then  And  A Given statement should only ever mention starting conditions. A When statement should always be some sort of verb, or action. A Then statement always has the words "should" or "should not", or equivalent. While not required, it is good practice for technical reasons to use "should" when implementing the automated test steps later.

The Scenarios are the most important part of the tests. Take time to make them well considered, granular, and create as many individual Scenarios as necessary to adequately test the feature in question. A good question to ask is "when this test fails, what will that failure tell us about the software?" The language of the Scenarios is reported directly in the output when the test fails, so it is important that each step in each Scenario be designed well. This is the heart of ATDD.

Note that anyone can contribute Scenarios at the "test backlog" page: http://www.mediawiki.org/wiki/Qa/test_backlog

Here are scenarios for a search test in use right now (from features/search.feature):

Feature: Search

Scenario: Search suggestions Given I am at random page When I search for: main Then a list of suggested pages should appear And Main Page should be the first result

Scenario: Fill in search term and click search Given I am at random page When I search for: ma     And I click the Search button Then I should land on Search Results page Feature files containing Scenarios are named "foo.feature" and reside in the /browsertests/features/ directory. Some good examples already exist.

Page Objects
Just as Cucumber implements an ATDD design pattern, we are using the Ruby gem called page_object to implement a design pattern called Page Object. The idea of a Page Object is that every element on every page in the test suite is defined once and only once, in the context of the page being tested. The simplest result of using page objects in browser tests is that it makes for ease of maintenance, since any change to how an element on a page is defined is changed in one and only one place.

Such a design allows more radical changes as well. For example, it would be possible to swap out entire systems, say mobile for desktop, or a different language for English, by changing only the page objects and not the actual tests.

An example page object for a simple search of Wikipedia looks like (from features/support/pages/search_page.rb): class SearchPage include PageObject

button(:search_button, id: 'searchButton') text_field(:search_input, id: 'searchInput') div(:search_results, class: 'suggestions-results') div(:one_result, class: 'suggestions-result') end "SearchPage" is our own name for the page being tested. "PageObject" tells the page_object gem what to do with the code that follows. Each element in the test is identified for that object for the purposes of the test. The general form to identify an element using the page_object gem is: (:, : ' ') Because we are using the watir-webdriver API for Selenium, it is possibly to specify more than one attribute for a single element if desired.

NOTE: it is good practice to list the page elements alphabetically by . This is for readability and maintainability, as large Page Objects can contain quite long lists of elements.

Comprehensive documentation on accessing elements with page_object is available on github: https://github.com/cheezy/page-object/wiki.

Page Object files are named "foo_page.rb" and reside in the /browsertests/features/support/pages/ directory.

Test Steps
With a Scenario (or several) in place, and a Page Object defined for the page to be tested, now we can tie them together with test steps. This is the programming part, and it is helpful to have some grasp of the technologies being used.

An implementation of a Search test using the Page Object above looks like (from features/step_definitions/search_steps.rb): Given /^I am at random page$/ do visit RandomPage end

When /^I click the Search button$/ do on(SearchPage).search_button end When /^I search for: (.+)$/ do |search_term| on(SearchPage).search_input= search_term end

Then /^a list of suggested pages should appear$/ do on(SearchPage).search_results_element.when_present.should exist end Then /^I should land on Search Results page$/ do @browser.url.should match '&title=Special%3ASearch$' end Then /^(.+) should be the first result$/ do |page_name| on(SearchPage).one_result.should == page_name end Each line in the .feature file becomes part of a regular expression in each test step. Just as we specify each element in each page only once and we specify each page only once, we also specify any particular aspect of the test such as text only once.

Since many tests may use a random page, RandomPage is specified separately. Because each line in the .feature file becomes a regular expression, we can using matching within the expression to specify strings to be searched for in the step, from the .feature file. We use a regex so we specify that text in the .feature file, and not in the steps file.

Note that the lines "Given I am at random page" and "When I search for:" are used twice in the .feature file. In the test steps, those lines are implemented once and called twice. Any single test step is implemented only once and may be used as often as necessary by the Feature. In the example above, we need only one test step to search for "main" and to search for "ma" using a regex as noted above.

In terms of architecture, it is Cucumber that ties together the lines in the .feature file and the test steps. It is the page_object gem that provides the means to identify elements on pages. The page_object gem is aware of both watir-webdriver API syntax and also pure selenium-webdriver syntax, so either approach is acceptable.

And for assertions, Cucumber uses the assertion framework called RSpec. This is why every "Then" statement in the test steps must include a "should" or "should_not" clause: otherwise the test will always pass and never fail. These are simple examples, but RSpec provides rich and complex ways to create potentially elaborate expectations for tests to satisfy beyond "should" and "should_not".

Comprehensive documentation for Cucumber and RSpec is beyond the scope of this document, but is readily available.

Configuration and Environment
Certain aspects of configuration and environment will be important to those who run and write these tests.

In the /features/support/ directory is a file called env.rb. Two aspects of the contents of this file are particularly important.

Many tests will require that the test negotiate the login dialog. In env.rb we point to user names and passwords for logging in. User names are generally public and reside in the file /config/config.yml. But passwords and certain other information should not be made public, and this information resides in a file /private/wmf/secret.yml. In order to run tests locally, the /private/wmf/secret.yml file is required. However, unless a particular test requires secret information, it is fine that secret.yml be empty. Adding information to config.yml is a public process, but if your test requires information like passwords or ssh keys, you will need to contact an administrator for help. Here is the section of env.rb that refers to config.yml and secret.yml: config = YAML.load_file('config/config.yml') mediawiki_username = config['mediawiki_username']

secret = YAML.load_file('/private/wmf/secret.yml') mediawiki_password = secret['mediawiki_password']

Running the Tests

 * clone the repository: git clone ssh:// @gerrit.wikimedia.org:29418/qa/browsertests
 * follow the instructions in README.md to configure Ruby for your system
 * create the file /private/wmf/secret.yml
 * in the browsertests/ directory, type 'bundle exec rake' to run all tests (note that tests requiring a password will fail until secret.yml contains the proper passwords)
 * to run a single test, in the browsertests/ directory type "bundle exec cucumber features/foo.feature" where "foo.feature" is the name of the feature test to be run.
 * the default browser will be Firefox. Alter the browser by changing local_browser in /features/support/env.rb

Further Information
This document does not describe every aspect of the design and architecture of the browser automation system. It is intended to explain enough that any interested person might get a good understanding of how things are put together and how to get started. The administrators of the project are happy to elaborate on aspects of the system not covered here.