Selenium/Ruby/Writing tests


 * To get started with browser testing, see How to contribute.

Test automation provides channels for non-programmers and inexperienced programmers to be valuable contributors. Non-programmers, particularly people involved in the product and project side of software development, can contribute well designed acceptance tests for features. Less experienced people who know the Document Object Model and browser behavior can contribute page objects for the automated tests. And more experienced programmers can contribute test steps that tie together acceptance tests and page objects by automating browsers.

Acceptance Test Driven Development
ATDD is a practice where we specify the behavior of software features in plain language sentences, often in some specific format. We then transform these specifications into executable code and threat them as automated acceptance tests for that feature. When the acceptance tests pass, the feature is known to be done. ("Done" has a specific meaning in agile software development: that the code is fit for use and ready to deploy.)

The canonical example of ATDD has the automated acceptance tests created before the code to satisfy them is created. In practice, this requires a level of cooperation between project/product people and development/test people that many organizations find difficult to achieve.

The current practice at WMF is to develop the feature specifications for Wikipedia software after or as the software is developed. There is nothing intrinsically wrong with this approach, although it is generally desirable to have testing activity match development activity as closely as possible, even to the point of moving testing activity *ahead* of development activity. Doing true ATDD is a great aid to focus and completeness of features.

In practice, acceptance tests for software features may be created and automated at any time, after which they serve as useful automated regression tests for those features. This is the point where non-programming members of the community, people with an interest in the proper function of Wikipedia software, can contribute.

We are using the Cucumber ATDD framework for browser automation. Cucumber requires that feature tests be specified as a series of statements of the form Given When  Then  People from any area of the Wikipedia community are potential contributors of acceptance test criteria for features. Wikipedia editors would certainly like to be assured that their favorite tools in their favorite browsers work and continue to work in the ways that they expect; Wikipedia user advocates such as the Teahouse contributors and Ambassadors likewise. WMF staff are also invested in the behavior of software features, with perhaps the Product area being the most concerned. Finally, there is a community of people outside Wikipedia who are interested in browser testing in and of itself who are a significant potential source of automated acceptance tests.

Designing and defining excellent acceptance test criteria in the form of browser behavior can be tricky, and we aim to train our community to do it well.

How To Run and Write WMF Browser Tests
Automated browser tests involve: You'll also need to watch out for aspects of the test environment.
 * 1) Features and Scenarios
 * 2) Page Objects
 * 3) Test steps

Potential contributors who are not programmers will be most interested in the description of Features and Scenarios. If you want to dig a bit deeper into how this project works, look at Page Objects. People actually creating end-to-end tests will be particularly interested in information about Test steps. And to run tests locally, learn the requirements for the local test environment.

To run the tests locally, see our instructions.

All code is in Gerrit (qa/browsertests) and a more readable mirror is at GitHub (wikimedia/qa-browsertests).

Features and Scenarios
Features and Scenarios in Cucumber are an implementation of a design pattern for Acceptance Test Driven Development, ATDD. They follow a generally accepted convention for such tests: Feature: My Feature Scenario: Testing some aspect of My Feature Given When  Then  Any Given/When/Then statement may also have an arbitrary number of "And" clauses: Scenario: Testing some complicated aspect of My Feature Given And When  And  And  Then  And  A Given statement should only ever mention starting conditions. A When statement should always be some sort of verb, or action. A Then statement always has the words "should" or "should not", or equivalent. While not required, it is good practice for technical reasons to use "should" when implementing the automated test steps later.

The Scenarios are the most important part of the tests. Take time to make them well considered, granular, and create as many individual Scenarios as necessary to adequately test the feature in question. A good question to ask is "when this test fails, what will that failure tell us about the software?" The language of the Scenarios is reported directly in the output when the test fails, so it is important that each step in each Scenario be designed well. This is the heart of ATDD.

Note that anyone can contribute Scenarios at the test backlog page.

Here are scenarios for a search test in use right now (from features/search.feature):

Feature: Search Scenario: Search suggestions Given I am at random page When I search for: main Then a list of suggested pages should appear And Main Page should be the first result Scenario: Fill in search term and click search Given I am at random page When I search for: ma     And I click the Search button Then I should land on Search Results page Feature files containing Scenarios are named "foo.feature" and reside in the /browsertests/features/ directory. Some good examples already exist.

Page Objects
Just as Cucumber implements an ATDD design pattern, we are using the Ruby gem called page_object to implement a design pattern called Page Object. The idea of a Page Object is that every element on every page in the test suite is defined once and only once, in the context of the page being tested. The simplest result of using page objects in browser tests is that it makes for ease of maintenance, since any change to how an element on a page is defined is changed in one and only one place.

Such a design allows more radical changes as well. For example, it would be possible to swap out entire systems, say mobile for desktop, or a different language for English, by changing only the page objects and not the actual tests.

An example page object for a simple search of Wikipedia looks like (from features/support/pages/search_page.rb): class SearchPage include PageObject

button(:search_button, id: 'searchButton') text_field(:search_input, id: 'searchInput') div(:search_results, class: 'suggestions-results') div(:one_result, class: 'suggestions-result') end "SearchPage" is our own name for the page being tested. "PageObject" tells the page_object gem what to do with the code that follows. Each element in the test is identified for that object for the purposes of the test. The general form to identify an element using the page_object gem is: (:, : ' ') Because we are using the watir-webdriver API for Selenium, it is possibly to specify more than one attribute for a single element if desired.

NOTE: it is good practice to list the page elements alphabetically by . This is for readability and maintainability, as large Page Objects can contain quite long lists of elements.

Comprehensive documentation on accessing elements with page_object is available on github: https://github.com/cheezy/page-object/wiki.

Page Object files are named "foo_page.rb" and reside in the /browsertests/features/support/pages/ directory.

Test Steps
With a Scenario (or several) in place, and a Page Object defined for the page to be tested, now we can tie them together with test steps. This is the programming part, and it is helpful to have some grasp of the technologies being used.

An implementation of a Search test using the Page Object above looks like (from features/step_definitions/search_steps.rb): Given /^I am at random page$/ do visit RandomPage end When /^I click the Search button$/ do on(SearchPage).search_button end When /^I search for: (.+)$/ do |search_term| on(SearchPage).search_input= search_term end Then /^a list of suggested pages should appear$/ do on(SearchPage).search_results_element.when_present.should exist end Then /^I should land on Search Results page$/ do @browser.url.should match '&title=Special%3ASearch$' end Then /^(.+) should be the first result$/ do |page_name| on(SearchPage).one_result.should == page_name end Each line in the .feature file becomes part of a regular expression in each test step. Just as we specify each element in each page only once and we specify each page only once, we also specify any particular aspect of the test such as text only once.

Since many tests may use a random page, RandomPage is specified separately. Because each line in the .feature file becomes a regular expression, we can using matching within the expression to specify strings to be searched for in the step, from the .feature file. We use a regex so we specify that text in the .feature file, and not in the steps file.

Note that the lines "Given I am at random page" and "When I search for:" are used twice in the .feature file. In the test steps, those lines are implemented once and called twice. Any single test step is implemented only once and may be used as often as necessary by the Feature. In the example above, we need only one test step to search for "main" and to search for "ma" using a regex as noted above.

In terms of architecture, it is Cucumber that ties together the lines in the .feature file and the test steps. It is the page_object gem that provides the means to identify elements on pages. The page_object gem is aware of both watir-webdriver API syntax and also pure selenium-webdriver syntax, so either approach is acceptable.

And for assertions, Cucumber uses the assertion framework called RSpec. This is why every "Then" statement in the test steps must include a "should" or "should_not" clause: otherwise the test will always pass and never fail. These are simple examples, but RSpec provides rich and complex ways to create potentially elaborate expectations for tests to satisfy beyond "should" and "should_not".

Comprehensive documentation for Cucumber and RSpec is beyond the scope of this document, but is readily available.

Configuration and Environment
Certain aspects of configuration and environment will be important to those who run and write these tests.

In the /features/support/ directory is a file called env.rb. Two aspects of the contents of this file are particularly important.

Many tests will require that the test negotiate the login dialog. In env.rb we point to user names and passwords for logging in. User names are generally public and reside in the file /config/config.yml. But passwords and certain other information should not be made public, and this information is thus defined in an environment variable named MEDIAWIKI_PASSWORD. In order to run tests locally, export a value for that environment variable; the value is the password to go with the user name defined in config.yml. For most Linux/Unix shells, the relevant command to define the environment variable will be: export MEDIAWIKI_PASSWORD=

Adding information to config.yml is a public process, but if your test requires information like passwords or ssh keys, you will need to contact an administrator for help. Here is the section of env.rb that refers to config.yml and secret.yml: config = YAML.load_file('config/config.yml') mediawiki_username = config['mediawiki_username']

Running VisualEditor browser tests via Vagrant
Some of the browser tests have been re-located to the specific code repositories for which they are most relevant. An example is browser tests for Visual Editor, which can be found in the VisualEditor GitHub repo. To run these tests in the Vagrant virtual environment, a couple of extra steps are necessary in order to make the "cucumber" script itself findable. Here are specific steps that should do the trick:

(in the directory where you have cloned the vagrant repo):


 * 1) Verify that local MediaWiki instance is running sucessfully via: http://127.0.0.1:8080/wiki/Main_Page
 * 1) Verify that local MediaWiki instance is running sucessfully via: http://127.0.0.1:8080/wiki/Main_Page
 * 1) Verify that local MediaWiki instance is running sucessfully via: http://127.0.0.1:8080/wiki/Main_Page

Further Information
This document does not describe every aspect of the design and architecture of the browser automation system. It is intended to explain enough that any interested person might get a good understanding of how things are put together and how to get started. The administrators of the project are happy to elaborate on aspects of the system not covered here.