Wikimedia Labs

Wikimedia Foundation engineering activity

Wikimedia Labs

A test environment using virtual machines for testing and experimentation

Group:	Operations
Start:
End:
Team:
Lead:	Ryan Lane
Status:	See updates

Wikimedia Labs is a two-part project aimed at helping volunteers get involved in Wikimedia operations and software development. The first part of this project is Test/Dev Labs, and the second part is Tool Labs.

Introduction to the toolserver and to Wikimedia Labs by Ryan Lane and Daniel Kinzler at the San Francisco Hackathon January 2012.

Background

Wikipedia's infrastructure and software was built and originally completely administered by community volunteers. As the site has grown over time the operations infrastructure and software development has been dominated by Wikimedia Foundation staff. While the Foundation's increased involvement has had a positive effect, we feel the trend of less volunteerism and more staff is something we'd like to reverse. The goal of Wikimedia Labs is to more easily enable involvement from our community by creating a virtualized environment that anyone can use to make build or make changes to infrastructure or to make changes to our software, demo them, have them code reviewed, and have them pushed to production.

It can also be used for small projects that are Wikimedia-related but distinct from the software used in production. For example, a project might present information from the MediaWiki API of a WMF project in a new way.

Access

You can create a Labs account by creating an account on wikitech.

Having a Labs account gives you access to Labs, Gerrit (our code review system) and a few other developer related tools.

Implementation

The architecture is described on Wikitech. The software for controlling this environment is implemented as a MediaWiki extension, and is described on mediawiki.org.

Status

• [edit status] • [add new]
  2013-04-05:
  Database replication for Wikimedia Labs work in progress (details)

Roadmap

For the roadmap, please see Wikimedia Labs's goals and planned milestones. See also Tool Labs/Roadmap en.

Open tasks

We'd love help with all of the below!

Development and sysadmin tasks

• Enable database replication - Asher hopes to get this done by January or February 2013
• Add request queues on the project pages
  • Having it embedded on that help page would be even better
  • We have some request queues right now, just not one for getting access to projects: https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests -- check out https://wiki.toolserver.org/view/Account_approval_process as a request queue that is simple and to be emulated
• Enable network-node per compute-node
• Enable IPv6
• Integrate RT with Labs LDAP
• Create a second zone in eqiad
• User databases
  • if we're going to hit user databases by end of March 2013, we likely need to put some dev work into salt; salt-api was pushed to GitHub mid-September, so some of the work has begun on their side, but we need to add keystone auth to that API
• Push OpenStackManager changes to show SSH fingerprints for instances
• Add DNS support to Nova
  • We added support for this in essex, but there's now an OpenStack project called Moniker we'll use instead. Waiting for it to enter incubation.

Product tasks

TODO: Add tasks

Proposals

• Development Process
• Puppet learning mode
• Toolserver features needed / wanted in Tool Labs (contains even some features which are not on toolserver)
• Investigate wikitech:Pentaho
• Find/test OTRS replacement, or upgrade/puppetize OTRS (with security patches)
• Puppetize PDF server
  • Package mwlib dependencies
  • Create an init script (preferably upstart) for mwlib daemon
    • Mentions of init script here (wikitech) but last I heard not in use. Peachey88 05:38, 21 December 2011 (UTC)
• Add puppet syntax highlighting to vim
• Create shared sql service for all projects
• Package and puppetize lilurl for use as a url shortening service
• Write Documentation for console
• Fix puppet repo so that it runs a complete first run of the puppet catalog on instance creation for all services
• Java App stack
  • Package JDK 1.6 (confused about this, can we not use openjdk or sun's packaged jdk?)
  • Apache Ant
  • Maven
  • Tomcat and Jetty App Servers
  • Apache Solr
  • Hadoop
  • Sbt
• Per-project saltstack remote execution
• Integrate a global queuing system (to replace tools' use of SGE when they migrate from the toolserver)
• Account creation improvement project
• Instance creation improvement project
• Interface usability improvement project
• Stability improvement project
• Communication improvement project

Completed

• Shared home directories per project
• Package adminbot, with an init script
• Puppet repository branch per project or instance
• Nagios management without exported puppet resources
• Package gerrit
• Create a log bot for the #wikimedia-labs channel
• Reverse proxy for web services

Documents

• Status updates

• Terms of use
• Agreement to disclosure of personally identifiable information
• Account creation text
• Things to fix in beta

Communications

• #wikimedia-labs^connect IRC channel on Freenode
• labs-l mailing list
• inventory of Labs's total capacity as of late September 2012 (compute, database and storage nodes)