Selenium/Ruby/Writing tests


 * To get started with browser testing, see How to contribute.

Test automation provides channels for non-programmers and inexperienced programmers to be valuable contributors. Non-programmers, particularly people involved in the product and project side of software development, can contribute well designed acceptance tests for features. Less experienced people who know the Document Object Model and browser behavior can contribute page objects for the automated tests. And more experienced programmers can contribute test steps that tie together acceptance tests and page objects by automating browsers.

Acceptance Test Driven Development
ATDD is a practice where we specify the behavior of software features in plain language sentences, often in some specific format. We then transform these specifications into executable code and threat them as automated acceptance tests for that feature. When the acceptance tests pass, the feature is known to be done. ("Done" has a specific meaning in agile software development: that the code is fit for use and ready to deploy.)

The canonical example of ATDD has the automated acceptance tests created before the code to satisfy them is created. In practice, this requires a level of cooperation between project/product people and development/test people that many organizations find difficult to achieve.

The current practice at WMF is to develop the feature specifications for Wikipedia software after or as the software is developed. There is nothing intrinsically wrong with this approach, although it is generally desirable to have testing activity match development activity as closely as possible, even to the point of moving testing activity *ahead* of development activity. Doing true ATDD is a great aid to focus and completeness of features.

In practice, acceptance tests for software features may be created and automated at any time, after which they serve as useful automated regression tests for those features. This is the point where non-programming members of the community, people with an interest in the proper function of Wikipedia software, can contribute.

We are using the Cucumber ATDD framework for browser automation. Cucumber requires that feature tests be specified as a series of statements of the form Given When  Then  People from any area of the Wikipedia community are potential contributors of acceptance test criteria for features. Wikipedia editors would certainly like to be assured that their favorite tools in their favorite browsers work and continue to work in the ways that they expect; Wikipedia user advocates such as the Teahouse contributors and Ambassadors likewise. WMF staff are also invested in the behavior of software features, with perhaps the Product area being the most concerned. Finally, there is a community of people outside Wikipedia who are interested in browser testing in and of itself who are a significant potential source of automated acceptance tests.

Designing and defining excellent acceptance test criteria in the form of browser behavior can be tricky, and we aim to train our community to do it well. For more information, see Writing feature descriptions.

How to write WMF browser tests
Automated browser tests involve: You'll also need to watch out for aspects of the test environment.
 * 1) Features and Scenarios
 * 2) Page Objects
 * 3) Test steps
 * 4) API-based setup methods

Potential contributors who are not programmers will be most interested in the description of Features and Scenarios. If you want to dig a bit deeper into how this project works, look at Page Objects. People actually creating end-to-end tests will be particularly interested in information about Test steps.

To run tests locally, see Running tests.

Code is in Gerrit (qa/browsertests) and a more readable mirror is at GitHub (wikimedia/qa-browsertests). Many MediaWiki extensions have local browser tests, often in a  subdirectory.

Features and Scenarios
Features and Scenarios in Cucumber are an implementation of a design pattern for Acceptance Test Driven Development, ATDD. They follow a generally accepted convention for such tests: Feature: My Feature Scenario: Testing some aspect of My Feature Given When  Then  Any Given/When/Then statement may also have an arbitrary number of "And" clauses: Scenario: Testing some complicated aspect of My Feature Given And When  And  And  Then  And  A Given statement should only ever mention starting conditions. A When statement should always be some sort of verb, or action. A Then statement always has the words "should" or "should not", or equivalent. While not required, it is good practice for technical reasons to use "should" when implementing the automated test steps later.

The Scenarios are the most important part of the tests. Take time to make them well considered, granular, and create as many individual Scenarios as necessary to adequately test the feature in question. A good question to ask is "when this test fails, what will that failure tell us about the software?" The language of the Scenarios is reported directly in the output when the test fails, so it is important that each step in each Scenario be designed well. This is the heart of ATDD.

Here are scenarios for a search test in use right now (from features/search.feature):

Feature: Search Scenario: Search suggestions Given I am at random page When I search for: main Then a list of suggested pages should appear And Main Page should be the first result Scenario: Fill in search term and click search Given I am at random page When I search for: ma     And I click the Search button Then I should land on Search Results page Feature files containing Scenarios are named "foo.feature" and reside in the /browsertests/features/ directory. Some good examples already exist.

Page Objects
Just as Cucumber implements an ATDD design pattern, we are using the "page-object" Ruby gem to implement a design pattern called Page Object. The idea of a Page Object is that every element on every page in the test suite is defined once and only once, in the context of the page being tested. This makes for ease of maintenance, since any change to that page element only requires updating one line in browser tests.

Such a design allows more radical changes as well. For example, it would be possible to swap out entire systems, say mobile for desktop, or a different language for English, by changing only the page objects and not the actual tests.

An example page-object for a simple search of Wikipedia looks like (from features/support/pages/search_page.rb): "SearchPage" is our name for the page being tested. The code following "PageObject" identifies the elements that tests reference on this page – buttons, links, message boxes, etc. The general form to identify an element is:  (:, : " ") Because we are using the watir-webdriver API for Selenium, you can specify more than one attribute for a single element.

Page-object files are named foo_page.rb and reside in the support/pages/ features/support/pages subdirectory.

NOTE: for readability and maintainability, list the page elements alphabetically by .

Page-object documentation is on github, you'll mostly use the elements.

Test steps
With a Scenario (or several) in place, and a Page Object defined for the page to be tested, now we can tie them together with test steps. This is the programming part, and it is helpful to have some grasp of the technologies being used.

An implementation of a Search test using the Page Object above looks like (from features/step_definitions/search_steps.rb): Given /^I am at random page$/ do visit RandomPage end When /^I click the Search button$/ do on(SearchPage).search_button end When /^I search for: (.+)$/ do |search_term| on(SearchPage).search_input= search_term end Then /^a list of suggested pages should appear$/ do on(SearchPage).search_results_element.when_present.should exist end Then /^I should land on Search Results page$/ do @browser.url.should match '&title=Special%3ASearch$' end Then /^(.+) should be the first result$/ do |page_name| on(SearchPage).one_result.should == page_name end Each line in the .feature file becomes part of a regular expression in each test step. Just as we specify each element in each page only once and we specify each page only once, we also specify any particular aspect of the test such as text only once.

Since many tests may use a random page, RandomPage is specified separately. Because each line in the .feature file becomes a regular expression, we can using matching within the expression to specify strings to be searched for in the step, from the .feature file. We use a regex so we specify that text in the .feature file, and not in the steps file.

Note that the lines "Given I am at random page" and "When I search for:" are used twice in the .feature file. In the test steps, those lines are implemented once and called twice. Any single test step is implemented only once and may be used as often as necessary by the Feature. In the example above, we need only one test step to search for "main" and to search for "ma" using a regex as noted above.

In terms of architecture, it is Cucumber that ties together the lines in the .feature file and the test steps. It is the page_object gem that provides the means to identify elements on pages. The page_object gem is aware of both watir-webdriver API syntax and also pure selenium-webdriver syntax, so either approach is acceptable.

And for assertions, Cucumber uses the assertion framework called RSpec. This is why every "Then" statement in the test steps must include a "should" or "should_not" clause: otherwise the test will always pass and never fail. These are simple examples, but RSpec provides rich and complex ways to create potentially elaborate expectations for tests to satisfy beyond "should" and "should_not".

Comprehensive documentation for Cucumber and RSpec is beyond the scope of this document, but is readily available.

Timeouts
Tests take time to run, especially in a VM on SauceLabs across a network or on your own slow MediaWiki instance.

So tests may fail due to timeouts when waiting a few more seconds would allow them to complete successfully. Inserting explicit  calls in tests is bad design. Instead, poll for conditions: All take an optional timeout parameter, the default is (???).
 * polls until an element can be engaged
 * polls until a condition returns true
 * polls until a condition returns false

That aft5 test is a good example:

This says: ''We'll hang out until the AFT post is processed. We know that the processing is finished when the page contains the text "Thanks". At that point we should have a message showing a link to the feedback page.''

Parallel tests
If you run tests in parallel with parallel cucumber and phantomjs you may speed up test time dramatically. These steps got CirrusSearch tests running in parallel:
 * Make sure all tests pass in PhantomJS. Mostly, this shouldn't require any work beyond installing PhantomJS, setting   to , and running tests.
 * Make sure your tests are in many small features rather than a few huge ones.4
 * Add  to your gemfile and run bundle install.

Running tests is now a two-step thing. First run tests in parallel. bundle exec parallel_cucumber --nice -n 5 features/ Some of them will fail, because of general flakiness. You can rerun failing tests like this: cat cucumber_failures.log | xargs bundle exec cucumber

API-based test setup methods
In order to allow for browser tests to run in a hermetic fashion (where they are not dependent on pre-existing wiki content), the mediawiki_api Rubygem has been developed. This Rubygem provides article and user creation methods, as well as an article deletion method, all based on the Mediawiki API. See the Rubygem documentation for further info on each available method and required input parameters.

To use these API methods in browser tests, include the Rubygem and relevant module in your env.rb file, as in the example shown below.

Then in your test steps file, call the relevant method that you would like to use, and send relevant parameters, as shown in the create_article example below.

In the steps file example above, parameters (title and content) are being passed to the step from an associated feature file. The relevant code in the associated feature file looks like the following (note that the example is not a complete feature file, but only shows relevant lines):

Further information
This document does not describe every aspect of the design and architecture of the browser automation system. It is intended to explain enough that any interested person might get a good understanding of how things are put together and how to get started. The administrators of the project are happy to elaborate on aspects of the system not covered here.