Wikimedia Engineering/Report/2012/May

 Engineering metrics in May:
 * 41 unique committers contributed code to MediaWiki.
 * The total number of unreviewed commits went from about 140 to 250.
 * About 34 shell requests were processed.
 * developers got developer access to Git and Wikimedia Labs, among which are volunteers.
 * Wikimedia Labs now hosts 97 projects, 177 instances and 431 users.

Major news in May include:
 * the publication of the [//blog.wikimedia.org/2012/05/11/book-architecture-mediawiki-open-source-applications/ Architecture of Open-Source Applications book], which contains a chapter on MediaWiki;
 * initial designs for a [//blog.wikimedia.org/2012/05/21/introducing-designs-for-the-universal-language-selector/ universal language selector];
 * a new and easier way to view a wiki's [//blog.wikimedia.org/2012/05/29/wikimedia-wikis-reveal-interwiki-map/ interwiki map];
 * [//blog.wikimedia.org/2012/05/29/1-million-media-files-uploaded-using-upload-wizard/ 1 million files] uploaded with our UploadWizard;
 * the [//blog.wikimedia.org/2012/05/31/wikidata-summit-kicks-off-in-berlin/ Wikidata/RENDER summit] in Berlin, followed by the [//blog.wikimedia.org/2012/06/02/diverse-wikimedia-tech-crowd-gathers-in-berlin/ hackathon].

Recent events
Berlin hackathon (1–3 June 2012, Berlin, Germany)

Upcoming events
Wikimania hackathon (10–11 July 2012, Washington, D.C., USA)

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.



Announcements

 * Danielle Benoit joined the Platform Engineering team as a contractor working on development tutorials (announcement).
 * Ori Livneh joined the Editor Engagement Experiments (E3) engineering team as Software Developer (announcement).
 * Vibha Bamba joined the Product team as Interaction Designer (announcement).
 * James Forrester joined the Product team as Technical Product Analyst, focusing on the Visual Editor (announcement).
 * Subramanya Sastry joined the Features team as Senior Software Engineer (announcement).

Site infrastructure
Data Centers
 * Month of May has been a busy month for racking, stacking and provisioning of newly purchased servers, especially by Rob Halsell, Chris Johnson, Leslie Carr and Mark Bergsma. Recently, we purchased new hardware for server refresh (many are out of warranty and over 4 years old), adding capacity and redundancy, and for new projects, including servers for  Search, Analytics, Fundraising, OpenStreetMap, databases, Varnish, Memcached and backups. Much effort was put into OS installation and servers network; they are now ready for the various systems and applications deployment.


 * IPv6 work went into full swing as well, in order to be ready for the June 6, IPv6 Launch Day. As of end of May, the database schemas were updated, and work started on refactoring LVS, PayBal, Varnish, Squid, DNS, Nagios monitoring and puppetization.


 * In April, we deployed the newly built Search cluster at our Ashburn datacenter, and disabled the Tampa search cluster. This month, Peter Youngmeister went through the exercise of upgrading the 4 years old Tampa search cluster infrastructure and brought it back up. We do have a cross-datacenter hot standby for 'Search' service now.


 * With Ubuntu 12.4 (Precise Pangolin) available, we have packaged and started using it selectively in some of our systems, including Search@Tampa and half of the LVS servers. Next we will be setting up the Apache servers at Ashburn data center using Precise as well.


 * Recently, we were experiencing a few systems rebooting themselves and while we did not see them impacting our services, Faidon Liambotis investigated and reported a bug with our kernel in 10.4 that caused our servers to reboot after about 208 days of uptime. As a result, we applied the necessary kernel and security patches to those impacted servers.


 * Ben has been working with the SwiftStack folks on enhancing SWIFT to provide additional SWIFT-specific monitoring to our ganglia tool. Here is the first preview - http://ganglia.wmflabs.org/latest/?r=20min&cs=&ce=&c=swift&h=i-000001c7.pmtpa.wmflabs&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS . Next, they will work on identifying potential SWIFT performance bottlenecks (when under load) in our implementation and recommend mitigation. Ben has started testing the upgrading of our current version to the just released 1.4.8. This should improve stability of the software.

Testing environment
Wikimedia Labs
 * The Labs infrastructure experienced several episodes of severe latency problems, all traced to the GlusterFS system. As a result, Ryan Lane, Faidon Liambotis and Andrew Bogott are working on a get well plan, which includes finding a suitable replacement of GlusterFS. The short term plan that is in the works however would expose us to a non-redundant infrastructure.

Backups and data archives
Data Dumps
 * We've been busy creating bundles of media in use per project and the first set of files is almost complete. For each wiki, there is now one or more files containing all media uploaded locally to the wiki, and one or more files containing all media used by the wiki but uploaded to Commons. We've also been preparing for the media back-end switch to Swift; since we won't be able to make copies of all media files in the usual way, some scripts were hacked together which will check the  and   tables and will retrieve and/or update media files via http as needed. Your.org and Masaryk University mirrors officially came online; we're still looking for other partners to host media backups and pageview statistics.

Internationalization and Editor Engagement Experiments
Editor Engagement Experiments (E3)

Wikidata

 * The Wikidata project is funded and executed by Wikimedia Deutschland.

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.