PHP is pretty much static/similar on all platforms, so running PHPUnit in CruiseControl or on your localhost is enough to know that the code is good (if it passes the test). -- This statement is mostly only of theoretical value, because recently we have seen some tests that were/are still broken on CC but not locally, or that were broken locally but not on CC.
This post was posted by Peachey88, but signed as 145.94.184.252.
Yes this is a problem. Some of it's differences in setup between ci.tesla and people's local installs. Differences in pass/fail between setups is usually an indication of a problem that needs fixing anyway :)
May also be related to CruiseControl install using SQLite and (most local installs) using MySQL.
I see that there is discussion on the page about having various browsers running in virtual machines.
A relatively cheap solution for this is VirtualBox, an open source project under GPLv2 for virtualization run by Oracle. This is an easy way to create VMs for Windows and various flavors of Linux (OS X is forbidden by Apple licensing requirements).
Some groups have been using VirtualBox in conjunction with Vagrant, which can deploy VirtualBox VMs through command line and/or script to set up test environments easily and then to take them down again.
I'm told that there may be licensing restrictions on using more than a set body of VirtualBox instances from Oracle but it isn't clear to me what the strictures might be and there are no docs about it on the website that I've seen.
As part of the Continuous integration project, Chad did some research to alternatives for CruiseControl.
I doesn't affect this RFC directly since Jenkins (the replacement that Chad found for CruiseControl) meets the same requirements and specification, it's just "better" and easier to use than CruiseControl.
Just thought I'd mention that here :)
I am not sure whether we should drop Selenium completely. Selenium tests do test the application as a whole. While I agree that a lot of things that are currently tested with Selenium can be substituted with QUnit or similar, I still think that selenium tests should be used to check more complex interaction patterns, e.g. to test a file uploader or GUI elements that interact via AJAX with the server.
Also, keep in mind that selenium tests are quite easy to record. While experienced developers like us probably tend to write the tests instead of recording, I think that we can actually reduce the barrier for others.
Both QUnit and TestSwarm are written with asynchronous and ajax support in mind. Our current test suite is in a static html-file, we're currently finishing up a to instead run these tests on Special:QUnitTest or something like that. But that's something on our end, not in QUnit.
File uploads are more complicated though, yes. But that is integration testing (or smoke testing), not unit testing. For unit testingæ' we want to and already have dropped Selenium. For integration testing, unless we find a better alternative, we should probably stick to Selenium.
Anyway, one thing at a time. Right now we're trying to clean up and get an infrastructure going on for Unit testing. Feel free to start an RFC for integration/smoke testing. There have been some alternatives, but it's an interesting topic to take next.
A community member requests a document that fulfills the following:
so I'm just noting that here so people can tell me "that already exists! LINK" or improve existing documentation accordingly.
Don't need to (unless running local setups)
Peachey88 wrote:
which tools contributors can trigger themselves
- None (Unless they are self hosting/running the tests on their boxes)
- CC automatically does the PHPunit tests
- Krinkle manages the TestSwarm on the TS (JS tests)
I'd make that instead:
I think this might be slightly over-stating the problems globals cause. Well we should certainly try to decrease their presence, I don't think they're preventing newbies from contributing, or causing huge problems in developing mediawiki (although they might affect tests more than normal development).
Personally, I don't see that much of a difference between a global like $wgOut that is pretty much the only instance of its respective class and a static method to get an instance of the class to be used with the request ( RequestContext::getMain()->getOutput() or whatever)
(Of course things like $wgTitle is just evil and needs to die, but that goes without saying ;)
The problem with global state objects is that people can (and do) change them on a whim. By incapsulating them in accessors, you keep that sort of thing from happening and behavior is much more testable.
Here follows probably the worst and unlikely example.....
... consider a newbie creating a new extension and doing some temporary variable assignments before doing stuff:
$wgIn = array( 'Awesome' );
function wfMakeOut( $in ) {
// Magic
return $in;
}
$wgOut = wfMakeOut( $wgIn );
foreach( $wgOut as $out ) {
$wgHooks['Foobar' . $out][] = 'wfMyFunc';
}
I know $wgOut isn't available yet in LocalSettings! like I said, bad example.
I think you have undersold the benefits of having unit tests. Here are two more reasons to have unit tests: Unit tests document exactly how the author expects the unit to be used. Providing known working examples of use makes it easier for someone else to learn to use that unit. My personal favorite reason is that, when it comes time to revise the unit (if it stays, you will), you have a way of assuring yourself that you have not broken the externally visible behavior. I like this because it helps relieve that knot in my stomach when I have to make changes to old code.
I agree there are many advantages / positive side effects of having unit tests.
My personal favorite advantage to writing unit tests, is discovering bugs when throwing things at units and seeing how it responds. It's not uncommon to commit changes to units during the creation of unit tests (ie. adding return statements to a "set" function).
It also acts a bit as "intensive code review". Writing unit tests (in my opinion) is most strong when done for methods written by someone else (ie. better to have someone else write unit tests for your modules). I don't think it should be a guide line though (it's fine to write your own unit tests), but when you have the chance and are in a team, it's worth considering writing unit tests for each other. That way it's basically an intensive code review, understanding of code base and bug hunt – all three at the same time.