Continuous integration/Task management

Server requirements

 * Chad, Timo
 * Packages needed are now listed here: Continuous integration/Overhaul
 * Packages needed are now listed here: Continuous integration/Overhaul

Getting server

 * Chad
 * Get a server, and puppetize everything.
 * Server has been assigned (gallium in eqiad), networking setup. Waiting on initial OS install from RobH.
 * Need to talk with Ops about how to puppetize the Server requirements
 * Done in 2011-12.
 * Done in 2011-12.

Publish

 * Chad, Timo
 * Install on the physical server and make available on http://swarm.testing.mediawiki.org and http://jenkins.testing.mediawiki.org 
 * On hold, needs server to be ready and the software to be ready. See sections below.
 * This is setup at [//integration.mediawiki.org integration.mediawiki.org]. Jenkins is available at [//integration.mediawiki.org/ci/ /ci] and TestSwarm at [//integration.mediawiki.org/ci/ /testswarm].
 * This is setup at [//integration.mediawiki.org integration.mediawiki.org]. Jenkins is available at [//integration.mediawiki.org/ci/ /ci] and TestSwarm at [//integration.mediawiki.org/ci/ /testswarm].

CruiseControl alternative

 * Chad
 * Research alternatives
 * We're going to go with Jenkins (formerly Hudson). For well-supported FLOSS CI systems that support PHP builds, the only two options are really CruiseControl and Jenkins. We've used CC for awhile now, and while it works, it lacks some of the features Jenkins provides us. Jenkins also has a larger community, a plethora of plugins to choose from, and abilities to tie into CR tools like Gerrit and deployment processes via SSH (automated fenari deployments after merge to wmf branch??). It also supports LDAP, so we can tie it in with SVN and the VM cluster.
 * We're going to go with Jenkins (formerly Hudson). For well-supported FLOSS CI systems that support PHP builds, the only two options are really CruiseControl and Jenkins. We've used CC for awhile now, and while it works, it lacks some of the features Jenkins provides us. Jenkins also has a larger community, a plethora of plugins to choose from, and abilities to tie into CR tools like Gerrit and deployment processes via SSH (automated fenari deployments after merge to wmf branch??). It also supports LDAP, so we can tie it in with SVN and the VM cluster.

MediaWiki development
Certain changes, improvements or new features have to be added to MediaWiki to support all this.

JavaScript unit testing

 * Timo
 * Issue tracker: bug 30339
 * Implement a new SpecialPage in core . By default the SpecialPage is disabled on a default install (since tests could be destructive (like making edits, watching things, make log actions). It can run one of several frameworks, for now just focus on QUnit . The run page loads QUnit + all test suites. Registry should be accessible by extensions as well.
 * First shot is almost complete. Moving into core and attaching to bug as patch for review. Krinkle 12:51, 13 August 2011 (UTC)
 * Branched Krinkle patch, reviewed it and patched it. See CodeReview for path /branches/JSTesting. Antoine &#34;hashar&#34; Musso 14:05, 20 October 2011 (UTC)
 * Made a point on bug 30339 for Krinkle. Apart from this, the branch seems good for a merge. Antoine &#34;hashar&#34; Musso 14:05, 20 October 2011 (UTC)
 * Merged to trunk in Krinkle 02:14, 6 January 2012 (UTC)
 * Merged to trunk in Krinkle 02:14, 6 January 2012 (UTC)

TestSwarm development
A number of improvements and bug fixes have to be made to the TestSwarm software in order to scale for our installation. As well as certain special needs for clean up (since we're not unit testing a static html file anymore but dynamic php).

Part of the plans that the jQuery Testing Team is making include work to be done on TestSwarm. Progress can be following on their wiki: http://jquerytesting.pbworks.com/

Timo is also a member of this team as of last July 2011.

Note that the below sections are not needed on the short term, but they are must-haves in the long run (say 1 year+)

User Agents

 * Timo
 * The database table for user agents, the maintainability of it and the way it influences future and past jobs isn't ideal. This should be rewritten and the jQuery Unit Testing team is also looking for a new system for this. This is an opportunity to collaborate.
 * The database table for user agents, the maintainability of it and the way it influences future and past jobs isn't ideal. This should be rewritten and the jQuery Unit Testing team is also looking for a new system for this. This is an opportunity to collaborate.

Caching

 * Timo
 * Several parts of TestSwarm don't scale very well and become either slow or are disabled by force. Implementation details of a querycache-table (or another solution) is being discussed.
 * Several parts of TestSwarm don't scale very well and become either slow or are disabled by force. Implementation details of a querycache-table (or another solution) is being discussed.

Bug fixes

 * Timo
 * About a dozen small but annoying bugs have been found and fixed in the Toolserver instance over time. These need to be reidentified, properly filed in the jQuery TestSwarm bugtracker and applied to our new install cleanly and submit patch upstream.
 * Some of the bugs fixed:
 * Table builder on "job"- and "run"-pages (previously ran over array of arrays, some of which contain more members than other causing array to be broken. Fixed by looping over the arrays first and changing from numeral array to associate array and then outputting by column)
 * Centered layout
 * Customizable site title
 * Customizable project navigation on top and "TestSwarm powered by + bug tracker links" at the bottom.
 * Customizable project navigation on top and "TestSwarm powered by + bug tracker links" at the bottom.

TestSwarm installation/configuration

 * 1) Find bugs, report bugs, fix bugs, patch upstream, pull request, (repeat).

Set up a vanilla install

 * 13:03, 13 August 2011 (UTC)
 * Timo
 * ''A basic unpatched install should be maintained during development. One that is just periodically updated from the remote HEAD of TestSwarm's git repository. A separate install is also on the server that runs our fork, which hopefully will be the same (just a little ahead awaiting their code review and processing of patches)
 * Fresh install here: http://ci2.tesla.usability.wikimedia.org/testswarm/

Automated MediaWiki checkout/installation

 * Timo, Chad, Hashar
 * We need to come up with a solid plan as to how we're going to do this. On every commit the script should check out MediaWiki, "install" it (LocalSettings, Database, etc.), and then submit jobs to the TestSwarm containing runs for all modules in all the supported skins. Aside from that we should think well about security (since the install needs to be somewhat public).
 * 2011-10-28: specification ongoing (ContinousIntegration-TestSwarmScript) –hashar, krinkle
 * 2011-12: This is done and deployed. Scripts live in SVN, [ ./trunk/tools/testswarm].
 * 2011-12: This is done and deployed. Scripts live in SVN, [ ./trunk/tools/testswarm].

Clean up

 * Timo
 * Garbage collection (delete old mw-installs and sqlite databases), but not the test results!.
 * Before we do so we need to implement support in TestSwarm for a "disabled"-state of a 'job' that should no longer distributed and locked (i.e. not allow swarm-admins to reset that revision for re-testing since the install would no longer exist). Krinkle 20:43, 28 October 2011 (UTC)
 * Probably a little database (one table?) in which we keep track of each revision (the revision id, the files on-disk that need deletion, and the job-id in TestSwarm that needs it's disable-state toggled). Krinkle 20:43, 28 October 2011 (UTC)
 * Probably a little database (one table?) in which we keep track of each revision (the revision id, the files on-disk that need deletion, and the job-id in TestSwarm that needs it's disable-state toggled). Krinkle 20:43, 28 October 2011 (UTC)

TestSwarm integration in Jenkins

 * Krinkle, Hashar
 * We want to do a few things in Jenkins, basically replacing TestSwarm's UI and using it only as a distributor Krinkle (talk) 20:50, 5 March 2012 (UTC)
 * We want to do a few things in Jenkins, basically replacing TestSwarm's UI and using it only as a distributor Krinkle (talk) 20:50, 5 March 2012 (UTC)

QUnit integration in Jenkins

 * Hashar
 * Since Jenkins is going to be the main entry point to verify MW functionality, it would be great to include the JS test suite in it. It is possible to have Jenkins run our QUnit test suite using Node.JS. It requires a few patch to MediaWiki javascript files and setup node-qunit (a port of QUnit under node.JS). All the code is pretty experimental though and need a few patches upstream and an unstable version of Node.JS. Antoine &#34;hashar&#34; Musso 14:13, 20 October 2011 (UTC)
 * On hold till upstream is stabilized enough. Maybe for Q1 2012? Antoine &#34;hashar&#34; Musso 14:13, 20 October 2011 (UTC)
 * Although it's interesting to have results of JS tests available in Jenkins, I'm not sure running them through Node.JS is the way to go. We could add it later but it's not a priority (especially since MediaWiki doesn't have to work in Node.JS, and several tests are probably useless anyway (the kind of tests that do DOM manipulation and animations). The jQuery Testing team is working on a plugin for Jenkins that allows TestSwarm to push or Jenkins to pull data from the TestSwarm database into Jenkins. Read more here. Setting that up is probably more useful right now then intergration with QUnit directly (thus skipping an actual browser). Note that we need first :) Krinkle 20:04, 20 October 2011 (UTC)
 * I'm proposing to cancel this task. The new test suite is no longer just plain html/javascript but runs in the MediaWiki environment including PHP at all. Which makes it a little more complicated to "quickly run the tests also from Node instead of a browser". Also, we no longer need this now that we can use the BrowserStack API and have real browsers in our swarm. And testing NodeJS as a goal on itself is pointless in my opinion as MediaWiki is written for the browser. Krinkle (talk) 20:36, 5 March 2012 (UTC)
 * I'm proposing to cancel this task. The new test suite is no longer just plain html/javascript but runs in the MediaWiki environment including PHP at all. Which makes it a little more complicated to "quickly run the tests also from Node instead of a browser". Also, we no longer need this now that we can use the BrowserStack API and have real browsers in our swarm. And testing NodeJS as a goal on itself is pointless in my opinion as MediaWiki is written for the browser. Krinkle (talk) 20:36, 5 March 2012 (UTC)