Meetings/Test framework/2010-07-11

Test Framework Discussion Notes
Location: Wikimania 2010

Attendees: Ryan Lane, Markus Glaser, Benedikt Kämpgen, Siebrand Mazeland, Priyanka Dhanda

Short Term Goals

 * Formalize the test framework roadmap. Make this visible to the community. (Robla and Priyanka)
 * Documentation (Ryan Lane)
 * Document the Selenium Grid setup on a public wiki (both hardware and software).
 * Make all documentation visible to the openqa community for both feedback and as guidelines to others trying to do the same thing.
 * Make a visual representation of all test system interactions - Continuum, PHPUnit, Selenium Grid (Ryan Lane)
 * Work in small iterative steps and make these steps visible.
 * For example: The first step should be clear documentation and a working example of how to write a test, run it against the grid and see the results.


 * Define testing best practices.
 * Encourage following code conventions and exporting tests to PHP.
 * We should encourage these best practices, but not frown upon tests that don't comply. There should be a process of sanitizing then and pulling them into the codebase.


 * Investigate and integrate the Selenium Framework with Apache Continuum. Look into a code coverage tool.

Long Term Goals

 * Find a good reporting strategy for test failures, code convention violations, etc.
 * Figure out a process to let community members write IDE tests and then incorporate them into the code base.
 * Who should be is responsible for maintaining extension and core tests? Should it be the developer?


 * Trigger a subset of tests on each commit and run all tests at some longer interval.
 * Have requirements and tests closely tied together through the feature specification page.
 * Make it easy for a bug submitter to submit test cases. Same with feature requests.
 * This way bug fixers and feature developers can use the test cases as a guideline and improve it as they go.

Outstanding issues with the current Selenium Framework

 * Initial content on the wiki we're testing.
 * For now tests should be responsible for content generation and cleanup.
 * When the number of tests increase we can figure out a better way to bootstrap data.


 * Add a configuration flag to allow tests to be run anonymously (Currently it requires login).
 * Dynamically reconfigure the wiki being tested.
 * Finalize the structure of the tests in the code base.
 * Now the test classes live in /tests/selenium. Each class is a suite with multiple test cases.
 * We should be able to run tests by a tag name and specify dependencies for each test suite.


 * Test job queues.
 * This may be easier if we have a configuration to put a wiki in test mode. Ideally it should enable flushing queues through the framework.