Talk:Selenium/PHP/Selenium Framework

Test suite setup

After yesterday's (5-14-2010) audio meeting, I thought a bit more about some of the test suite start-up issues. Here are some random thoughts (note: my interest is regression testing, so that is the context of these comments):

• Originally, I thought it would be best to startup the selenium-server at machine boot. However, I no longer think this is the correct strategy. You probably want a separate selenium-server per user/client/test developer. Also, you really only need the server to run during a test. So, perhaps a better idea is to instrument the framework with a setup facility. This would be called before a test suite runs. One of the things it could do is test to see if the selenium-server is running and if not, start it.

• Another thing the setup facility could do is setup the database appropriately. This assumes a test suite is designed to run using common wiki data. If not (e.g., each test needs different data in the wiki), then you would need a per-test setup facility.

• Yesterday, someone mentioned the idea of setting up the test database by dropping tables and then repopulating them. Another idea is to drop the database and recreate it from an existing database (nb: our wiki uses postgres, so I am not sure of the implications of this idea for mysql databases).

Dnessett 19:28, 15 May 2010 (UTC)

Previous page history was archived for backup purposes at Talk:Selenium Framework/LQT Archive 1 on 2015-07-10.

How are we going to run selenium tests for extensions? As of right now, extension tests must be hard coded in. We need to figure out a way of integrating the tests automatically when MediaWiki is undergoing selenium testing.

Idea:

• Each extension has a hook that is called for adding/running tests.
• The hooks are only called when MediaWiki is configured to use Selenium.

On the test framework side, running tests for extensions can be done by adding the test suite name to the selenium_settings.ini. Eg:
testSuite[WikiEditorTestSuite] = "extensions/WikiEditor/tests/selenium/WikiEditorTestSuite.php"
On the test wiki side, seems like the setup needs to happen before the Setup.php. So instead of using hooks this is what I'm thinking of doing:

• LocalSettings.php should contain

require_once("extensions/WikiEditor/tests/selenium/WikiEditorSeleniumConfig.php");
$wgSeleniumTestConfigs['WikiEditorTestSuite'] = 'WikiEditorSeleniumConfig::getSettings';

WikiEditorSeleniumConfig::getSettings returns an array of include files and configuration variables needed by the extension.

• If a cookie has been set, indicating this is part of the WikiEditorTestSuite test run, we use the configuration we get from WikiEditorSeleniumConfig::getSettings

I'm doing some final testing and cleanup, and then I'll update the page and check in.

I only read the implementation details part, and it looks mostly sane to me. I don't think it's a good idea to have test suite names double as hook names; you could use "SeleniumTest$testsuite" or whatever to avoid scary naming collisions, but even then I think it'd be better to just call something like Selenium::doSetupFor( $testsuite ) (calling a hook makes no sense, because there'd be exactly one listener) and have that function know where to look. The latter could be implemented with a one-dimensional equivalent of $wgHooks (i.e. array( 'testsuite' => xallback ) ) if you like, just don't pollute $wgHooks itself with this that aren't really hooks.

I like how you're doing all this with a cookie, that's smart. It'll break in subtle ways if Squid caching is used (and in even more interesting ways if Selenium is hitting production URLs through Squid). It's possible to fix this, but as far as I'm concerned it can go in the "don't be stupid" category.

So after 1.14, MediaWiki started returning 404 status codes for pages that don't yet exist. Selenium seems to die when it encounters a 404. Are you all dealing with that in some way? For example, how do you test browsing to a page that doesn't exist and clicking the create link? --Cneubauer 13:42, 28 September 2010 (UTC)

I try to test SimpleSeleniumTestSuite so I execute :

php tests/RunSeleniumTests.php

• Error1 : PHP Fatal error: require_once(): Failed opening required 'PHPUnit/Runner/Version.php' (include_path='.:/usr/share/pear:/usr/share/php') in /home/karima/git/wikimedia-git-repos/core/tests/RunSeleniumTests.php on line 34
  • test (maybe) : pear install -f phpunit/PHPUnit_Selenium
  • result : may be ok but new error (Error2)
• Error2 : PHP Warning: Invalid argument supplied for foreach() in /home/karima/git/wikimedia-git-repos/core/tests/RunSeleniumTests.php on line 134
  • test : maybe the position of config file but with the argument seleniumConfig, I have same error

php tests/RunSeleniumTests.php --seleniumConfig /home/.../core/selenium_settings.ini

An idea ?

I found, there is error "foreach" because the parameter $seleniumTestSuites is NULL when [SeleniumTests] is empty in selenium_settings.ini.

ah, cool. Thanks for sharing :)