Wikimedia Engineering/Report/2012/July

 Engineering metrics in July:
 * The total number of unreviewed commits went from about 320 to about 360.
 * About 35 shell requests were processed.
 * About 80 developers got access to Git and Wikimedia Labs.
 * Wikimedia Labs now hosts projects,  instances and  users.

Major news in July include:

Recent events
Pre-Wikimania hackathon (10–11 July 2012, Washington, D.C., USA)

This year's pre-Wikimania Hackathon was special in that it had a full track for newcomers, going beyond tutorials. The Hackathon was a collaboration with OpenHatch, an open source teaching non-profit. The new efforts included identifying appropriate first-time tasks to orient newcomers into more advanced Wikipedia editing and tech contribution, creating a laptop setup guide that steps attendees through the process of configuring development environments, and providing constant in-person assistance to help people past problems they encountered. While at the event, we saw many people learning more about templates, editing Wikipedia, and using and modifying bots to improve the encyclopedia and media on it. At least 65 people signed in, with more surely more in attendance. A more full report on the Hackathon is forthcoming to the wikitech-l list.

Wikipedia Engineering Meetup
To showcase the interesting problems and products that Wikimedia engineering work on to the local developer community, the Tech group has created a Wikipedia Engineering Meetup. The Meetup plans meet every two months at WMF offices in San Francisco and tentatively consists of three short 15 minute engineering presentations followed by a question and answer period bracketed by mingling.

The inaugural meetup will be on August 15 and the talks scheduled will be:
 * Tomasz Finc and Jon Robson talking about Wikipedia Mobile
 * David Schonover talking about Analytics at the Foundation
 * Trevor Parscal and Roan Kattouw talking about the VisualEditor

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.



Announcements

 * Peter Youngmeister, who was working as a contractor for the Operations team, was converted to full-time Technical Operations Engineer (announcement).
 * S Page joined the Editor engagement experiments team as Software Engineer (announcement).
 * S Page joined the Editor engagement experiments team as Software Engineer (announcement).

Site infrastructure

 * July was a quiet month for Operations, and the team were working on mostly small and behind the scene projects. Mark has successfully tested the upgraded Varnish software with persistent cache feature on some of our mobile caching servers. They are working very well and the plan is to roll it widely in the coming week or two. Mark and Peter are completing the work on the new application server packaging and build. The beta version looks pretty stable during the tests. This build is running on Precise (Ubuntu 12.04) and will work on the Swift object store (rather than the current nfs filer). They will be performing further performance and scalability tests by rolling it out to a portion of Tampa application servers shortly.

Asher has deployed an upgraded version of the parser cache and the results have been impressive. Comparing p90 and p99 cache get times averaged over several days (July 3-5) for db40 vs the last 8 hour for pc1 shows p90 dropping from 53.6ms to 7.17ms, and p99 dropping from 185.3ms to 17.1ms. This is relevant to every page request from logged in and cookied logged out users so should have a meaningful impact on the user experience.

Data Centers

Object Store/Swift

Testing environment
Wikimedia Labs

Backups and data archives
Data Dumps
 * The YAS3 library for uploading to archive.org and to other s3-compatible sites, along with several command line clients, is now usable (though still under heavy development). This library handles 100 Continue correctly; this means that for large file uploads, the upload is only attempted once the client has been redirected to the right host, a great time saver. The library also supports uploads of large files in multiple chunks automatically, rather than requiring the user to split the file into separate pieces. That's a necessity for us since many of our dump files are quite large.

Offline
Kiwix


 * We finally released Kiwix 0.9 rc1 (see the CHANGELOG). All the binary files were compiled using our new continuous integration build platform. In collaboration with Wikimedia France (for the Afripedia project), we released a first version of kiwix-plug, a standalone WiFi hotspot using cheap plug computers. The Black&White project, contracted by Wikimedia CH, was completed; a recent achievement was the introduction of Kiwix in the official Debian package repository. Also in collaboration with Wikimedia CH, we started a new project called ZIM autobuild aiming to quickly and automatically generate ZIM files of our projects.

Wikidata

 * The Wikidata project is funded and executed by Wikimedia Deutschland.

The Wikidata team has made good progress towards their first roll-out. The initial deployment plans are being made and the Hungarian Wikipedia community stepped up to be the first to use the interwiki part of Wikidata in a few weeks. You can follow the deployment planning at Wikidata/Deployment. This also means the demo system needs to be tested more. If you have five spare minutes, have a look at the demo system and report any bugs you might find there so they can be fixed before the initial deployment.

The team also started to collect future use-cases of Wikidata that should be kept in mind during development. You can find the existing ones here and are invited to refine them or add your own. Additionally the team is looking for feedback on the third iteration of the storyboard for linking Wikipedia articles in the future.

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.