Wikimedia Labs

Wikimedia Labs is a two-part project aimed at helping volunteers get involved in Wikimedia operations and software development. The first part of this project is Test/Dev Labs, and the second part is Tool Labs. For the roadmap, please see Wikimedia Labs's goals and planned milestones.

You can create a Labs account to do MediaWiki development, tools, or analytics by simply filling out the account creation form.

Wikimedia Labs is a sandbox server environment, based on a virtualization cluster, for supporting development and operations engineering by both staff and volunteers. This will allow access to necessary system resources, software and configuration, without compromising the security of the production server clusters, and in a time- and cost-effective way.



Goals
There are three major goals of Test/Dev Labs:


 * 1) To improve collaboration between staff and volunteer developers.
 * 2) Have a process for providing higher levels of access for people who are not on the paid operations team staff. This includes staff developers, and all volunteers. We'd like to have an environment where anyone can eventually become root, even on our production cluster.
 * 3) Have an environment where we can test major changes before we deploy them to the live site.

Tool Labs
The design for Tool Labs is still in the very early phases; it may change drastically as the project ramp up begins in January 2013. The first target of Tool Labs is to be a replacement for Toolserver; see Wikimedia Labs/Toolserver features needed in Tool Labs.

There is now a (partial, in progress TODO list at Wikimedia Labs/TODO).

Goals

 * Provide an environment for rapid prototyping
 * Close to production, but simplified
 * Easy deployment of MediaWiki
 * Provide a ramp for new developers
 * Tools and extensions can easily move to Test/Dev Labs
 * Provide a location for bot authors to maintain and run their bots (in progress at bots.wmflabs.org)
 * Provide a location for analytics work

A key goal of Tool Labs is to provide an easy development environment meant to be used as a ramp to the Test/Dev Labs environment. The goal is to facilitate the rapid development of software that we can then identify as something target-able for production support. Software can be moved from Tool Labs to Test/Dev Labs, formalized, and then deployed to production.

Implementation
The architecture is described on Wikitech. The software for controlling this environment is implemented as a MediaWiki extension, and is described on mediawiki.org.

TODO
For the roadmap, please see Wikimedia Labs's goals and planned milestones (schedule).

We'd love help with all of the below!


 * Enable database replication - Asher hopes to get this done by January or February 2013
 * Add the sysadmin/netadmin membership to the projects page
 * It would also be nice for projects to have a request queue
 * Having it embedded on that help page would be even better
 * We have some request queues right now, just not one for getting access to projects: https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests -- check out https://wiki.toolserver.org/view/Account_approval_process as a request queue that is simple and to be emulated
 * Enable network-node per compute-node
 * Enable IPv6
 * Integrate RT with Labs LDAP
 * Create a second zone in eqiad
 * User databases
 * if we're going to hit user databases by end of March 2013, we likely need to put some dev work into salt; salt-api was pushed to GitHub mid-September, so some of the work has begun on their side, but we need to add keystone auth to that API
 * Push OpenStackManager changes to show SSH fingerprints for instances
 * Add DNS support to Nova
 * We added support for this in essex, but there's now an OpenStack project called Moniker we'll use instead. Waiting for it to enter incubation.
 * Add Gluster support to Nova (Gluster plugin is mostly written but we don't really have plugin support until Folsom, so will happen in Q3 when we are ready to upgrade to Folsom)

Proposals

 * Development Process
 * Puppet learning mode
 * Toolserver features needed / wanted in Tool Labs (contains even some features which are not on toolserver)
 * Install Pentaho
 * Find/test OTRS replacement, or upgrade/puppetize OTRS (with security patches)
 * Create a bot running infrastructure (partially done - test / "production" not done)
 * Puppetize PDF server
 * Package mwlib dependencies
 * Create an init script (preferably upstart) for mwlib daemon
 * Mentions of init script here (wikitech) but last I heard not in use. Peachey88 05:38, 21 December 2011 (UTC)
 * Reverse proxy for web services
 * Add puppet syntax highlighting to vim
 * Create shared sql service for all projects
 * Package and puppetize lilurl for use as a url shortening service
 * Write Documentation for console
 * Fix puppet repo so that it runs a complete first run of the puppet catalog on instance creation for all services
 * Java App stack
 * Package JDK 1.6 (confused about this, can we not use openjdk or sun's packaged jdk?)
 * Apache Ant
 * Maven
 * Tomcat and Jetty App Servers
 * Apache Solr
 * Hadoop
 * Per-project saltstack remote execution
 * Integrate a global queuing system (to replace tools' use of SGE when they migrate from the toolserver)
 * Account creation improvement project
 * Instance creation improvement project
 * Interface usability improvement project
 * Stability improvement project

Completed

 * Shared home directories per project (done)
 * Package adminbot, with an init script (done)
 * Puppet repository branch per project or instance (done)
 * Nagios management without exported puppet resources (done)
 * Package gerrit (done)
 * Create a log bot for the #wikimedia-labs channel (done)

Documents

 * Status updates


 * /Terms of use/
 * /Agreement to disclosure of personally identifiable information/
 * /Account creation text/
 * /Things to fix in beta/

Communications

 * IRC channel on Freenode
 * labs-l mailing list
 * inventory of Labs's total capacity as of late September 2012 (compute, database and storage nodes)