Wikimedia Engineering/Report/2013/June

Engineering metrics in June:
 * 116 unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits remained stable around 960.
 * About 30 shell requests were processed.
 * Wikimedia Labs now hosts 166 projects and 1520 users; to date 2052 instances have been created.
 * The tools project in Labs now hosts 222 tools and 193 members.

Major news in June include:

''Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.

Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Announcements

 * Sean Pringle joined the Technical Operations team as our Storage and Database Engineer (announcement)
 * Brian Wolff joined the Wikimedia Platform Engineering group as Software developer for the Summer, working on multimedia contribution and review (announcement).
 * Ken Snider joined the Technical Operations team as an international contractor, poised to fill the Director of Technical Operations position (announcement).
 * Toby Negrin joined the Engineering department as Director of Analytics (announcement)
 * Toby Negrin joined the Engineering department as Director of Analytics (announcement)

Technical Operations
Site infrastructure
 * As part of our capacity planning work, Mark upgraded most of our varnish infrastructure (EQIAD & ESAMS) with newer and faster servers.  He will be adding new mobile varnish servers in ESAMS next, this coming month. RobH and Daniel are pushing ahead with the migration of the other applications from Tampa to Eqiad, Documentation/updates on this matter can be found here  - ***https://wikitech.wikimedia.org/wiki/Tampa_cluster . New Parsoid application and varnish servers were also deployed in anticipation of the coming Visual Editor deployment.  Meantime, AlexK is starting the backup project work. Initial project writeup can be found on - https://wikitech.wikimedia.org/wiki/Projects#Backup_infrastructure  and the technology writeup is at https://wikitech.wikimedia.org/wiki/Bacula.
 * Mark put in the finishing touches to deploy all the new network infrastructure at ESAMS. With help from Leslie and Mark, we finally got approval from ARIN for some new IPv4 addresses. Those IP addresses are needed for our new ULSFO buildup.
 * Many people are refactoring puppet code with the ultimate goal of having everything organized into puppet modules. Andrew Bogott, Antoine and Alexandros are setting up an automated testing infrastructure to support these efforts.

Data Dumps
 * Our GSOC student, Petr Onderka, is set up in gerrit and has his first commits in on the Incremental Dumps project. You can look at his code here:, follow his progress here:  and you can check the current discussion on the mailing list here: .  Additionally, we have IRC meetings on weekdays at about 4:15 pm UTC in #wikimedia-tech; lurkers and contributors are welcome!

Wikimedia Labs
 * AJAX improvements for OpenStackManager deployed to wikitech (added actions: console output; improvements: reboot)
 * Added interface for displaying quotas for projects in OpenStackManager
 * Ensured all instances were properly running puppet and salt. Many instances were running puppetmaster::self and needed to have local puppet repo merges or rebases.
 * Upgraded salt everywhere and re-issued keys to fix a vulnerability in salt
 * Worked on stabilization of the NFS server. We've encountered a kernel bug with NFS and have changed the scheduler from cfq to deadline and have decreased the read and write sizes of clients to 8k.
 * Progress has been made towards making the Labs DB replicas available to the Labs at large (as opposed to only the Tool Labs project).
 * Much work has been done towards user request fulfillment in Tool Labs, including work towards WSGI support.

Kiwix
The Kiwix project is funded and executed by Wikimedia CH.

Development of a new Mediawiki HTML dumper in nodeJS has started. This solution exports Wikipedia articles in static files based on the Parsoid output. This solution is really promising, new javascript developers are welcome.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.

June in Wikidata was all about the sister projects. The development team published proposals for how Wikidata can support Commons and Wiktionary. Additionally they worked on the ability of Wikidata to store language links to Wikivoyage in addition to Wikipedia so that Wikivoyage will soon also be able to manage their language links via Wikidata. Another important step was the deployment of the geoccordinate datatype. With this it is now possible to indicate the location of a city for example. Geocoordinates that are already in Wikidata can be seen on this map (huge version - updated daily).

In a blog entry Denny explained his understanding of the relation of Wikidata and the truth.

In further news: further development of Wikidata has been supported through a large donation by the search engine company Yandex.

Future

 * The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.