Selenium/Ruby/Environment abstraction layer

Project Overview
The Environment Abstraction Layer component of the mediawiki_selenium framework was started as part of a effort to: 1) improve test determinism by enforcing an outside-in read-only environment configuration; 2) provide constructs around common MediaWiki-related resources that will simply test patterns; 3) better isolate test implementation that requires multiple sessions; and 4) allow for custom configuration of browser settings such as language, proxies, etc. without having to add support to the core framework.

Environmental Contract
We face limitations in the "given" steps of test implementation when it comes to setting up initial application state. Lacking the database access required for test fixtures we must, instead, rely on other methods of resource provisioning such as the MW API or Selenium-driven configuration via the UI. For resources that cannot be setup using either method—such as instances of wikis themselves and additional users (on account of CAPTCHA)—we rely on a handful of prerequisite environments that have those resources in place: Beta Cluster, Wikipedia Test2, and local instances of MediaWiki-Vagrant.

Because these environments can differ in the way they're setup, it's important the we be able to articulate to our test suite a consistent set of assumptions about the state of the resources therein. For instance, "this is the base wiki URL," "this is the sandboxed test user," "this is where you can make API calls," etc. In more advanced use cases, this can include things like "this is a new user, you can assume it has no edit history." In a way, these bits of configuration can be looked at as a contract between the environment and the test suite.

Configuration is currently accomplished using a simple set of environment variables that are referenced in code using Ruby's  global. This is sufficient for many use cases, but it doesn't quite have the contract-like qualities necessary to ensure deterministic test behavior. For one, there's no guarantee of failure when a requested variable isn't found and, secondly, the values of  variables are, well ... variable.

The EAL addresses these two issues directly by enforcing a strict configuration lookup and preventing any changes to its state at runtime.

Strict Lookup
Ruby's  global is a simple   that provides no guarantee or expectation of failure when a requested variable is not found. A developer can implement their own checks, but that must be done for each and every case, and many times leads to redundant nil/empty string checks.

If a strict check for the configuration is not enforced, it typically leads to unexpected/obscure failure down the line.

The EAL solves both of these problems by enforcing a uniformly strict policy on configuration lookup unless a default value is explicitly provided.

Protected State
Mutation of the global  object is permitted by Ruby, but it's generally not a good idea as it can lead to race conditions and other unpredictable behavior, especially in evented contexts such as Cucumber's.

To illustrate the complications, consider how we (for security reasons) configure the test-user's password in CI by way of a alternative variable (corresponding to ).

Now imagine this same global variable is referenced somewhere else, say in another  hook that's defined somewhere in the project's test-suite.

It's entirely possible that, depending on how the various support files were imported, this second hook would end up executing before the first, an example of a simple race condition.

The EAL prevents these conditions by providing a one-way, read-only path for environment configuration; configuration is sourced at the beginning of each run, may be read by step definitions, but is never defined or redefined at runtime.

Configuration Defaults
So the EAL's configuration can be considered a strict contract, but who is the authority on its correctness? We could leave it up the environment's test runner to define all the necessary configuration, but then we risk incorrect assumptions or changes in the environment affecting the outcome of our tests—this is exactly the kind of indeterministic behavior the EAL is meant to reduce.

If we consider environmental configuration to be as important to test determinism as the resources created in "given" context by calls to the API or by simulating UI interaction, we should probably allow the test suite to dictate at least a correct default configuration for different situations.

As of now, there's no support planned for the sourcing of a local configuration file directly in the EAL but, fortunately, Cucumber already provides a suitable feature: profiles.

Cucumber profiles are typically used to specify default  parameters, but it can also specify environment variables. Leveraging this feature, we can allow test suites to define their own defaults for each environment that's expected to run it.

Passwords are an exception to this for obvious reasons. They should always be omitted from files in the repo.

Rich Environment Resources
Beyond just codifying the way that environment configuration is defined and read, the EAL provides additional constructs around common MW resources so that more expressive and readable test implementation can be written.

It allows for steps like the following:

To be rewritten as:

A Whole New MediaWiki World
Every Cucumber step implementation runs in the context of a  instance which is created anew for each scenario. (In Ruby speak, the step-definition block is evaluated with  as the Cucumber   object, probably using  . ) This construct of a "world" object fits nicely with the idea of an "environment" interface and, fortunately, Cucumber allows us to customize the world object used.

Helper Methods
By setting up our own world object, we're not only neatly encapsulating our configuration at the beginning of each scenario, we're also making available all the instance methods of the  class. This allows us to provide some nice constructs that make for more readable tests and flexible tests. The most important of these helper methods are:


 * : Current wiki user name
 * : Current wiki user name (with underscores replaced)
 * : Current wiki user's password (taking into account )
 * : Current browser instance (started on demand)
 * : Access to the browser factory for setting up custom settings that last over the duration of the scenario
 * : Perform actions on another wiki
 * : Perform actions as another user
 * : Perform actions in an isolated (and optionally customized) browser session