Selenium/PHP/Selenium Framework

Purpose
SeleniumFramework provides a standardised way of writing Selenium tests for MediaWiki and its extensions. It is based on PHPUnit tests.

Prerequisites
In order to work properly, the framework builds on some external dependencies:
 * Java
 * Selenium server, either Selenium RC or Selenium Grid
 * Available at seleniumhq.org
 * Direct link: Selenium RC 1.0.3
 * phpunit (e.g. PEAR PHPUnit 3.4 library)
 * PEAR Testing_Selenium
 * pear install Testing_Selenium-0.4.3
 * (note: you can try "pear install Testing_Selenium", but it may give you an error if you don't specify the stable version)

Note: phpunit and PEAR Testing_Selenium must be accessible within your PHP path.

Configure the framework
Here's the short version:
 * Make a selenium_settings.ini
 * if you use PHP5.3 or later then
 * Copy maintenance/test/selenium/selenium_setting.ini.sample to $IP/selenium_settings.ini
 * if you use PHP5.2 then
 * Copy maintenance/test/selenium/selenium_setting.ini.php52.sample to $IP/selenium_settings.ini
 * Edit your selenium_settings.ini. At a minimum, you'll probably need to edit your wiki URL
 * Add the following to LocalSettings.php

// Basic configuration $wgEnableSelenium = true; $wgSeleniumConfigFile = $IP. '/selenium_settings.ini';

// A really simple example require_once("maintenance/tests/selenium/SimpleSeleniumConfig.php"); $wgSeleniumTestConfigs['SimpleSeleniumTestSuite'] = 'SimpleSeleniumConfig::getSettings';

// Another more complete example for WikiEditor (note: requires WikiEditor extension) require_once("extensions/WikiEditor/tests/selenium/WikiEditorSeleniumConfig.php"); $wgSeleniumTestConfigs['WikiEditorTestSuite'] = 'WikiEditorSeleniumConfig::getSettings';

The configuration above assumes a single wiki for everything. It is possible to have two separate wikis (one driving the tests, and one being tested). More information can be found in Selenium Configuration.

Writing tests

 * In order to add a Selenium test, you have to create a test suite that extends SeleniumTestSuite. For example look at maintenance/tests/selenium/SimpleSeleniumTestSuite.php
 * You have to implement the function addTests to add your test cases to the suite. You can override setUp and tearDown as required.
 * Then create as many test cases as you need and make them extend SeleniumTestCase. You can override setUp and tearDown as required.

Examples
[SeleniumTests] testSuite[SimpleSeleniumTestSuite] = "maintenance/tests/selenium/SimpleSeleniumTestSuite.php"
 * For a very basic sample test suite and test case look in maintenance/tests/selenium/SimpleSeleniumTestSuite.php and maintenance/tests/selenium/SimpleSeleniumTestCase.php
 * You will need the following in the selenium_settings.ini file to run these tests
 * A working example can also be found at the PagedTiffHandler extension.

Test Wiki configuration
Multiple test suites that need different wiki configurations are run against a single test wiki.

Requirements

 * The test wiki should have a default configuration. This is the state that test suites should assume for a test wiki before setUp is called.
 * Each test suite needs to tell the test wiki what configuration it needs. This may be
 * files to be included for an extension
 * global configuration variables
 * Could also be used to switch out the db and images directory.
 * (Do we need more fine grained control, like per test instead of per test suite?)


 * At the end of a test suite, the wiki needs to go back to its original default configuration.

Missing

 * Concurrent test suites point to the same wiki
 * Whether this will work to change the db and images directory for the test wiki.
 * If any extensions do custom configuration that is not an include or a global config variable.

Implementation details
Test suite: Test wiki: The following happens in WebStart.php only if a global variable $wgEnableSelenium is set.
 * The test suite's setUp makes a request to the test wiki with a request parameter whose value is the test suite name.
 * After this the test can assume that the the test wiki is configured correctly for this test suite.
 * The test suite's tearDown sends another request parameter which will tell the test wiki to do back to using the default configuration.


 * If request param setupTestSuite is set
 * set a cookie containing the test suite name
 * cookie is set to expire in 10 mins (Should this be configurable)


 * If request param clearTestSuite is set
 * clear the cookie containing the test suite name
 * If we find our testSuiteName cookie
 * Look for a global variable $wgSeleniumTestConfigs[testSuiteName]. If found, this is the test suite's function that is responsible for providing the following.
 * the test specific include files
 * the test specific globals (key value pairs)


 * $wgSeleniumTestConfigs[testSuiteName] needs to be set in LocalSettings.php. The reason this is not a done via a hook is because this configuration needs to happen before Setup.php.

Test wiki configuration example: As an example see extensions/WikiEditor/tests/selenium
 * In the case that it finds neither of the request params mentioned above nor the cookie, the Setup will proceed as usual with only the default configuration.
 * LocalSettings.php should contain
 * WikiEditorSeleniumConfig::getSettings returns an array of include files and configuration variables needed by the extension.
 * If a cookie has been set, indicating this is part of the WikiEditorTestSuite test run, we use the configuration we get from WikiEditorSeleniumConfig::getSettings

Architecture
These files are part of the framework:
 * RunSeleniumTests.php includes test suites specified in extension directories.
 * selenium/Selenium.php provides access to selenium and implements common tasks.
 * selenium/SeleniumLoader.php loads all the neccessary classes for the framework.
 * selenium/SeleniumTestSuite.php starts and stops selenium tests.
 * selenium/SeleniumTestCase.php provides some additional assertions.
 * selenium/SeleniumTestListerner.php is the interface to result logging.
 * selenium/SeleniumConsoleLogger.php and selenium/SeleniumHTMLLogger.php produce the actual output.
 * selenium/SimpleSeleniumTestSuite.php and selenium/SimpleSeleniumTestSuite.php provide an example test case.
 * Individual tests are located with the extensions in the directory selenium.

The tests should be located here:
 * Tests for core should be placed in the maintenance/tests/selenium folder
 * Extensions that do mirror the mw directory structure (like SMW) should place the tests similar to mw: extensions/EXTENSION/maintenance/tests/selenium
 * Extensions that don't should place the tests in extensions/EXTENSION/tests/selenium

Testing with a clean database and file state
testrunner                    wiki under test --                    --- 1.1 start selenium which in turn starts a browser to talk to the wiki under test 1.2 send request for new test with unique test id and tests that will be fired 2.1 create cookie with test id                              2.2 create temporal resources according to tests list 2.3 create test tracker with timestamp 2.4 return success code 3.1 start testsuites via selenium 3.2 send a lot of individual requests according to the tests 4.1 testrunner is identified by test id                              4.2 reconfigure database and resources according to test id                              4.3 ? Do something with memcached ? 4.4 execute request 4.5 update timestamp in test tracker 5.1 send a teardown request 6.1 execute teardown, i.e. delete all resources associated with test id                              6.2 delete test tracker 6.3 return success code 7.1 stop selenium

Test styleguides

 * A test should leave at best no traces in the wiki.
 * If this is not possible, the test should leave the wiki in a state which allows the test to be re-run.

Add common test tasks
Please add to this list...
 * Trigger user preferences
 * Edit a page (initial support already implemented)
 * Upload images (already in PagedTiffHandler tests)
 * Show edit and history mode
 * Revert pages

Add common assertions
Please add to this list
 * Text within an article
 * Text in Headlines
 * Text in Table of Contents
 * Error messages

Improvements

 * Make test result logging more talkative
 * Check prerequisites of MediaWiki configuration (initial support in PagedTiffHandler tests)
 * Find a way to execute selenes recorded via Selenium IDE
 * Refactor naming conventions
 * Ways to configure the wiki for specific test suites or extensions.
 * A way to bring the database to a clean initial state before each test
 * Use the mediawiki extension to write collaborative Selenium tests

Further ideas
A test (or a single test suite) should be able to reconfigure the wiki according to its needs. Here, some serious security issues arise. This might be a possible way: If the hook is not triggered in LocalSettings, no reconfiguration can take place.
 * Add a hook 'LocalSettingsEnd' at the end of LocalSettings.php
 * Add some URL parameter which indicates which test is being run
 * Within the extension, the hook code now changes the configuration

Error about Testing_Selenium (0.4.3 is installed from pear along with PHPUnit) not being found

 * Replace /PEAR/ by the actual path to the repository

General Tips

 * Ignore port A proxy time you out port 4444.
 * Uncomment any tests in RunSeleniumTests.php to run different test suites?

Experiences: Recent changes in handling of framework

 * PEAR PHPUnit 3.4 and PEAR Testing_Selenium: Check whether properly installed, esp. PHPUnit/Framework.php
 * Installation of selenium-rc is pretty easy:
 * Download selenium rc
 * start "java -jar selenium-server.jar"