Selenium/PHP/Test framework deployment

This is a project to get an integrated testing framework in place.
 * Markus Glaser added initial Selenium testing support to the repository,
 * Ryan Lane has been building a cluster,
 * Priyanka Dhanda has been refining the Selenium configuration
 * Mark Hershberger integrating with CruiseControl to run both Selenium and PHPUnit tests
 * Michelle Knight has been recording Selenium tests

Status: Permanent hiatus
Update 2013-12-31: Browser testing has been revived using cucumber: More details. So this page about test framework deployment has only archival function. --Mglaser (talk) 13:25, 31 December 2013 (UTC)

Update 2011-07-01: We sort of set up Selenium Grid, which automatically runs Selenium tests in different virtual environments (each with a different browser/OS etc.).

But most contributors ignored Selenium; they don't find its output useful. And Selenium Grid is an ops headache (it's a chore to maintain the VMs, and sometimes a Selenium Grid test failure actually indicates a Grid or virtual machine problem rather than a broken test!). Also, Selenium Grid's scope was limited: It remotely controls browsers in the VMs, so unless Selenium supported an OS and browser, it couldn't be part of the 'grid'.

WMF shut down their Selenium Grid VMs a few weeks ago in favor of running QUnit with Jenkins. Sumanah 18:56, 1 July 2011 (UTC)

Feature justification
Currently, we do not have a lot of formal testing of MediaWiki, except for some usability work that Wikimedia Foundation has worked with Calcey to provide. Automated testing has also been lacking.

Selenium provides a means of doing automated system testing. This work would be complementary to work on Apache Continuum, which can provide us the means of continuously running Selenium tests, and with PHPUnit, which provides us with a framework for unit testing.

User requirements
See Test framework deployment/Requirements

Specification

 * Documentation about the existing Selenium Framework (in progress)

Software design document

 * Documentation about the existing Selenium Framework (in progress)

Schedule
Milestones correspond to status meetings:
 * Sept 17 - send out proposed configuration (with prototype code)
 * Sept 24 - more or less working dynamic configuration, with a test or two actually using it
 * Oct 1 - dynamic config problem solved, and a couple of extension tests, initial test runs on the grid
 * Oct 8 - running better on the grid
 * Oct 15 - running smoothly on the grid
 * Oct 22 - finished

Task management
(e.g. link to relevant Bugzilla queries)

Automation work done by the Calcey team
Calcey has worked on basic automation of Wikimedia usability, using Selenium.

Test scripts are recorded using Selenium IDE 1.0.7 and converted to PHP. Test scripts were implemented such that data and code can be maintained individually and are reusable.

The automation suite can be found on the <> location. Recorded test scripts can be executed using the NetBeans IDE 6.8 which was integrated with PHP Unit and Selenium module for PHP.

Automated scripts can be run individually as separate scripts or complete test suite can be run at once. Test results will be generated with status of passed or failed for each and every test scenario available in the test scripts.

Functionality automated using Selenium.

1. Search functionality
 * Search for a wiki page and verify search results directed to the page
 * Search for a text and verify the search results contain the required text

2. Watch page
 * Mark a page as watch and verify the My Watch list
 * Mark a page as watch and then unwatch and verify the My Watch list
 * Mark a page as watch on page edit and verify the My Watch list

3. Adding header levels
 * Add header level 2, 3, 4 and 5 and verify header outputs
 * Add header levels 2 & 3  and verify header output
 * Add header levels  2, 3 & 4  and verify header output
 * Add header levels  2, 3,4 & 5  and verify header output

4. Editor
 * Add a internal link and verify the link on wiki preview
 * Add a internal link with different display text and verify the link on wiki preview
 * Add a internal link with blank display text and verify the link on wiki preview
 * Add a external link and verify the link on wiki preview
 * Add a external link with different display text and verify the link on wiki preview
 * Add a external link with blank text and verify the link on wiki preview
 * Add a table and verify the table
 * Add a table only with header row and verify the table
 * Add a table only with borders and verify the table
 * Add a table without headers, borders and sort rows and verify the table
 * Add a table with headers, borders and sort rows and verify the table
 * Verify the search and replace function
 * Verify Match Case option with ‘Replace All’ option
 * Verify regular expression option with ‘Replace All‘ option

5. Toolbar
 * Verify media wizard function and verify the output
 * Add reference file function and verify the output
 * Add picture gallery function and verify the output

6. Text format
 * Add a bold text and verify the output on the wiki preview
 * Add a italic text and verify the output on the wiki preview
 * Add both italic and bold text and verify the output on the wiki preview
 * Add bulleted items and verify the output on the wiki preview
 * Add numbered items and verify the output on the wiki preview
 * Add a no-wiki text and verify the output on the wiki preview
 * Add a line break and verify the output on the wiki preview
 * Add a big text and verify the output on the wiki preview
 * Add a small text and verify the output on the wiki preview
 * Add a superscript text and verify the output on the wiki preview
 * Add a subscript text and verify the output on the wiki preview