Wikimedia Engineering/Report/2013/June

Engineering metrics in June:
 * 116 unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits remained stable around 960.
 * About 30 shell requests were processed.
 * Wikimedia Labs now hosts 166 projects and 1520 users; to date 2052 instances have been created.
 * The tools project in Labs now hosts 222 tools and 193 members.

Major news in June include:

''Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.

Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Announcements

 * Sean Pringle joined the Technical Operations team as our Storage and Database Engineer (announcement).
 * Brian Wolff joined the Wikimedia Platform Engineering group as Software developer for the Summer, working on multimedia contribution and review (announcement).
 * Ken Snider joined the Technical Operations team as an international contractor, poised to fill the Director of Technical Operations position (announcement).
 * Toby Negrin joined the Engineering department as Director of Analytics (announcement).

Technical Operations
Site infrastructure
 * As part of our capacity planning work, Mark Bergsma upgraded most of our Varnish infrastructure (in EQIAD & ESAMS) with newer and faster servers. He will be adding new mobile Varnish servers in ESAMS next, this coming month. Rob Halsell and Daniel Zahn are pushing ahead with the migration of the other applications from Tampa to EQIAD. New Parsoid application and Varnish servers were also deployed in anticipation of the coming VisualEditor deployment. Meantime, Alexandros Kosiaris is starting the backup project work; read more about the project and the technology.
 * Mark also put in the finishing touches to deploy all the new network infrastructure at ESAMS. With help from Mark and Leslie Carr, we finally got approval from ARIN for some new IPv4 addresses, needed for our new ULSFO buildup.
 * Many people are refactoring Puppet code with the ultimate goal of having everything organized into Puppet modules. Andrew Bogott, Antoine Musso and Alexandros are setting up an automated testing infrastructure to support these efforts.

Data Dumps
 * Our GSoC student, Petr Onderka, is set up in gerrit and committed his first contributions to the Incremental Dumps project; you can follow his code, read his progress reports and check the current discussion on the mailing list. Additionally, we hold IRC meetings on weekdays at about 4:15 pm (UTC) in #wikimedia-tech; lurkers and contributors are welcome.

Wikimedia Labs
 * Wikimedia Labs saw a lot of improvements in June, including the deployment of AJAX improvements for OpenStackManager to wikitech (added actions: console output; improvements: reboot), and a new interface for displaying quotas for projects in OpenStackManager. We ensured that all instances were properly running Puppet and Salt; Many instances were running puppetmaster::self and needed to have local puppet repo merges or rebases. We upgraded Salt everywhere and re-issued keys to fix a vulnerability in Salt. The team also worked on stabilizing the NFS server. We've encountered a kernel bug with NFS; we have changed the scheduler from cfq to deadline and have decreased the read and write sizes of clients to 8k. Progress has been made towards making the Labs database replicas available to the Labs at large (as opposed to only the Tool Labs project). Last, much work has been done towards user request fulfillment in Tool Labs, including work towards WSGI support.

Kiwix
The Kiwix project is funded and executed by Wikimedia CH.

Development of a new MediaWiki HTML dumper in nodeJS has started. This tool exports Wikipedia articles in static files based on the Parsoid output. This solution looks really promising, and new JavaScript developers are welcome.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.

June in Wikidata was all about the sister projects. The development team published proposals for how Wikidata can support Commons and Wiktionary. Additionally, they worked on the ability of Wikidata to store language links to Wikivoyage in addition to Wikipedia; as a result, Wikivoyage will soon also be able to manage their language links via Wikidata. Another important step was the deployment of the geocoordinate datatype. This makes it possible, for example, to indicate the location of a city. Geocoordinates that are already in Wikidata can be seen on this map (huge version, updated daily).

In a blog entry, Denny Vrandečić explained his understanding of the relation of Wikidata and the truth.

In other news: further development of Wikidata has been supported through a large donation by the search engine company Yandex.

Future

 * The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts. Annual goals for the 2013–2014 fiscal year are currently being drafted.