Selenium/Ruby/Environment abstraction layer

Project overview
The Environment Abstraction Layer component of the mediawiki_selenium framework was started as part of a effort to: 1) improve test determinism by enforcing an outside-in read-only environment configuration; 2) provide constructs around common MediaWiki-related resources that will simply test patterns; 3) better isolate test implementation that requires multiple sessions; and 4) allow for custom configuration of browser settings such as language, proxies, etc. without having to add support to the core framework.

Environmental contract
We face limitations in the "given" steps of test implementation when it comes to setting up initial application state. Lacking the database access required for test fixtures we must, instead, rely on other methods of resource provisioning such as the MW API or Selenium-driven configuration via the UI. For resources that cannot be setup using either method—such as instances of wikis themselves and additional users (on account of CAPTCHA)—we rely on a handful of prerequisite environments that already have those resources in place: Beta Cluster, Wikipedia Test2, and local instances of MediaWiki-Vagrant.

Because these environments can differ in the way they're setup, it's important the we be able to articulate to our test suite a consistent set of assumptions about the state of the resources therein. For instance, "this is the base wiki URL," "this is the sandboxed test user," "this is where you can make API calls," etc. In more advanced use cases, this can include things like "this is a new user, you can assume it has no edit history." In a way, these bits of configuration can be looked at as a contract between the environment and the test suite.

Configuration is currently accomplished using a simple set of environment variables that are referenced in code using Ruby's  global. This is sufficient for many use cases, but it doesn't quite have the contract-like qualities necessary to ensure deterministic test behavior. For one, there's no guarantee of failure when a requested variable isn't found and, secondly, the values of  variables are, well ... variable.

The EAL addresses these two issues directly by enforcing a strict configuration lookup and preventing any changes to its state at runtime.

Strict lookup
Ruby's  global is a simple   that provides no guarantee or expectation of failure when a requested variable is not found. A developer can implement their own checks, but that must be done for each and every case, and many times leads to redundant nil/empty string checks.

If a strict check for the configuration is not enforced, it typically leads to unexpected/obscure failure down the line.

The EAL solves both of these problems by enforcing a uniformly strict policy on configuration lookup unless a default value is explicitly provided.

if empty end

Immutable state
Mutation of the global  object is permitted by Ruby, but it's generally not a good idea as it can lead to race conditions and other unpredictable behavior, especially in evented contexts such as Cucumber's.

To illustrate the complications, consider how we (for security reasons) configure the test-user's password in CI by way of an alternative variable (corresponding to ).

Now imagine this same global variable is referenced somewhere else, say in another  hook that's defined somewhere in the project's test-suite.

It's entirely possible that, depending on how the various support files were imported, this second hook would end up executing before the first, an example of a simple race condition.

The EAL prevents these conditions by providing a one-way, read-only path for environment configuration; configuration is sourced at the beginning of each run, may be read by step definitions, but is never defined or redefined at runtime.

Configuration defaults
So the EAL's configuration can be considered a strict contract, but who is the authority on its correctness? We could leave it up the environment's test runner to define all the necessary configuration, but then we risk incorrect assumptions or changes in the environment affecting the outcome of our tests—this is exactly the kind of indeterministic behavior the EAL is meant to reduce.

If we consider environmental configuration to be as important to test determinism as the resources created in "given" context by calls to the API or by simulating UI interaction, we should probably allow the test suite to dictate at least a correct default configuration for different environments.

Via cucumber profiles
One way to to provide defaults for environment variables would be through Cucumber profiles. They are typically used to specify default  parameters, but can also specify environment variables. Leveraging this feature, we could allow test suites to define their own defaults for each environment that's expected to run it.

Via a native feature
The approach using Cucumber profiles may have some downsides, however. For one, there's no way to specify a different set of defaults for different environments; in non-default environments, the user would have to explicitly provide a profile on the command line. Secondly, there's no way to allow overwriting of the variable values; profiles don't support any kind of variable substitution like.

A native approach might provide more flexibility. Change Ifb358ec20 provides a mechanism by which you can define these environmental defaults in a file named   within browser-test directory. It's similar to the Cucumber profiles configuration in that it's a YAML hash where each key corresponds to a environment name.

The set of defaults that should be used is specified as a single  variable.

Some of the key (seemingly advantageous) differences here are: 1) the format of  is a bit more congruent with how variables are referenced in the step implementation; 2) the values can be overwritten by variables defined in the shell; and 3) there's no extra invocation option necessary as long as the user has   correctly set in the shell.

Rich environment resources
Beyond just codifying the way that environment configuration is defined and read, the EAL provides additional constructs around common MW resources so that more expressive and readable test implementation can be written.

It allows for steps like the following:

To be rewritten as:

A whole new MediaWiki world
Every Cucumber step implementation runs in the context of a  instance which is created anew for each scenario. (In Ruby speak, the step-definition block is evaluated with  as the Cucumber   object, probably using  . ) This construct of a "world" object fits nicely with the idea of an "environment" interface and, fortunately, Cucumber allows us to customize the world object used.

Helper methods
By setting up our own world object, we're not only neatly encapsulating our configuration at the beginning of each scenario, we're also making available all the instance methods of the  class. This allows us to provide some nice constructs that make for more readable and flexible tests. The most important of these helper methods are:


 * : Current wiki user name
 * : Current wiki user name (with underscores replaced)
 * : Current wiki user's password (taking into account )
 * : Current wiki URL
 * : Perform actions on another wiki
 * : Perform actions as another user
 * : Perform actions in an isolated (and optionally customized) browser session
 * : Current browser instance (started on demand)
 * : Access to the browser factory for setting up custom settings that last over the duration of the scenario

Switching between resource alternatives
As you can see, some of the helper methods allow for performing actions on "other" wikis or as "other" users. Indeed, the EAL has a construct for switching between "alternative" configurations. This serves test cases where one has to simulate interactions between multiple users, across multiple wiki, etc. (The Echo and Flow extensions require these kinds of scenarios, for example.)

At first this may seem to contradict the idea that configuration should be static from start to finish, but the EAL actually accomplishes the temporary re-configuration in a very isolated way: by cloning the  object, overwriting its config with the alternative values, and evaluating the given block within the new object's context. In other words, the new environment only affects the scope of the block and the changes don't amount to a change in global state that would threaten us with race conditions and other unpredictable behavior—this is actually a fairly common pattern used in functional programming.

Alternative configuration values are looked up using the base name (e.g. ) appended with the alternative ID (e.g.  ).

For example, given the following configuration:

When the following step is evaluated:

I expect  and   within the block to be.

Note that while the above example illustrates how alternative configuration is resolved and overwritten, a more realistic scenario involving two wikis would probably use  (which shares implementation with  ).

Reducing coupling with names over values
One of the aims of the EAL is to reduce environment coupling, like in this scenario:

Scenario text (and user stories in general) is most useful as acceptance criteria when it includes only what is relevant to the feature's requirements. In other words, is it important in this case we're logged in as "Selenium_user"? No, not really. Is it important that we're clicking the "Preferences" link? Yes. Therefore, we'd probably want to refactor this scenario to read.

Without the EAL, you would probably implement the first step using the value for the  environment variable (directly using  ), which is simple enough. However, without the EAL, this sort of refactoring doesn't work in cases where we're describing more than one of the same kind of resource, for example the interaction between two users.

In order to reduce coupling in cases like this, one can make use of the EAL's named alternatives. The above example could be refactored as the following scenario text and step definitions.

As you can see, the actual name of the user is factored out which reduces coupling and, furthermore, the logic between scenario steps and implementation remains clear due to the use of descriptive variables.

Isolated browser sessions
MediaWiki developers of user-to-user features like Echo and Flow often need to implement scenarios that simulate multiple user sessions, which can be problematic for a number of reasons. Most notably, logging out (before logging back in as someone else) destroys all sessions of the current user, including those of concurrently running tests.

The EAL attempts to solve issues like these by providing an easy way to instantiate multiple browsers within the context of a single scenario.

As noted previously, whenever possible, you typically want to provision the initial state described by the  clauses using the API. Assuming that's not an option here—ostensibly, Flow topics cannot be created via the API, but if that's not actually the case, just pretend it is for the sake of this example—we'd have to resort to driving the UI to create the initial topic; we'd want to be logged in as the default user to create the initial topic, then as user "B" to respond to it, then again as the default user to perform the  and measure the.

This would be quite cumbersome in the current framework, requiring some sort of tag, a before hook, and an after hook to close out the extra browser session. Use of page objects also becomes problematic: they expect a  instance variable to be set, so you might have to repeatedly overwrite it.

With the EAL's  helper method (in combination with other helpers like  ), we can easily perform any number of operations within the context of a new session, which is started on demand, cached, and closed automatically at the end of the scenario.

In the example above, we've opened up a new browser session, and named it "b", to perform authentication and subsequent actions of user "b". If we wanted to perform additional actions in separate steps using this same browser, we could again reference it using.

Another useful aspect of  is that custom browser settings can be given as the second argument.

More on the implications of customizing or changing browser settings in browser factories.

Browser factories
(TODO)