Wikimedia Engineering/Report/2011/August

Major news in August include:
 * Technical discussions at Wikimania and Developer days;
 * Progress on HTTPS, and generally better processes in operations;
 * The kick-off of the Internationalization and localization tools project;
 * New features in UploadWizard, including major work on customized campaigns for the Wiki Loves Monuments event;
 * MediaWiki 1.18 and the new mobile platform approaching deployment readiness;

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * Wikimania (August 2–7, Haifa, Israel) — A delegation of about a dozen Wikimedia engineers attended the Wikimania conference in early August, as well as the Developer Days and OpenZIM Developers Meeting. A special focus of the dev days was the recruitment and mentoring of new MediaWiki developers. Extensive notes were taken, with the hope of turning them into perennial on-wiki documentation.


 * Google Tech Talk (August 25, Mountain View, California) — Erik Möller, Rob Lanphier and Alolita Sharma presented a tech talk at Google to give an all-round update across Wikimedia's engineering projects, to help refresh the understanding of Googlers and other interested parties. The presentation slides are available, and the video will be made available through the GoogleTechTalks Youtube channel.

Upcoming events

 * New Orleans hackathon (14-16 October, New Orleans) – Ryan Lane and Sumana Harihareswara are organizing a coding event on the theme "The infrastructure of innovation". The hackathon's goal is to advance Wikimedia's tools and infrastructure; a major focus will be Wikimedia Labs, starting with the dev-ops virtualization cluster. Other areas of work include gadgets/extensions/tools support, authorization/authentication strategy, and general training and hacking.
 * Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. The following positions opened in August:
 * Product Manager (New Editor Engagement)

New and open requests for proposals:
 * Engineering Outreach
 * Development and Operations Engineer
 * Article Feedback Feature

The following positions are still open: Product Manager (Analytics), QA Lead, Operations Engineer (Networking), Director of Features Engineering, Systems Engineer (Data Analytics), Software Developer (Rich Text Editing, Features), Software Developer (Front-end) and Software Developer (Back-end), Product Manager (Mobile), Software Developer (Mobile)

Short news

 * Jeremy Postlethwaite joined as Software Engineer for Fundraising (announcement).
 * The team for the Internationalization and localization tools (I18n/L10n) project was put together in August. It consists of Siebrand Mazeland (Product Manager, Localization), Niklas Laxström & Santhosh Thottingal (I18n/L10n Software Engineers), and Gerard Meijssen (I18n/L10n Outreach Consultant).
 * Contractor Aaron Schulz is now a full-time employee, working as Software Developer Back-end (announcement).

Training and process improvement

 * Operations staff meeting — The operations team got together the week of the 22nd. The goals of the meeting were to: improve and share site recovery knowledge (documentation and training); share knowledge of new project designs; review & prioritize operations projects; document and communicate our EQIAD data center buildup milestones; and to develop the RT management process.

Site infrastructure

 * Tampa data center — Mark Bergsma put into production the second router, which means we have now router redundancy in our Tampa network infrastructure. Follow-up work is underway to fully implement auto-router (hot) failover, which should be completed by mid September. Mark also standardized our LVS implementation and puppetized the configuration. Since our data center contractor in Tampa left us, the installation of new application servers was delayed. Other highlights in August include software upgrade to our Squid servers, and upload performance issues (now solved).


 * Virginia data center — Asher Feldman deployed the new Mobile Varnish servers in our Eqiad data center. All six LVS servers are ready and two of them are in production now, load-balancing the mobile Varnish servers. Ben Hartshorne and Asher also created two new database servers for the Summer of Research interns, in addition to the original one created last month. Last, the team updated the backup procedures documentation to reflect Eqiad being our (current) key backup and recovery store.


 * Media storage — Russ Nelson refactored code in the SwiftMedia extension following Tim Starling's code review feedback; he also asked users for peculiar use cases involving file manipulation. Ben Hartshorne started to plan for the deployment and switchover on Wikimedia sites.


 * HTTPS — Roan Kattouw and Ryan Lane have been fixing issues that surfaced during the internal testing period. New servers were ordered for the AMS and TPA to handle SSL termination processing. In the meantime, Ryan has been setting up SSL servers in eqiad, enhancing Varnish to deal with X-Forwarded-For and X-Forwarded-Proto HTTP headers, and making necessary changes to Squid. On August 31st, HTTPS was enabled on wikimediafoundation.org and Wikimedia Commons.

Testing environment

 * Virtualization test cluster — Work has resumed on this project. The puppet configuration used in the production sites has been split into public and private repositories, and all sensitive information has been moved to the private repository. Gerrit has been configured, and the public puppet configuration will soon be moved into a public repository there. Labs LDAP and SVN LDAP are currently being merged, so that SVN users will more easily have access to Labs.

Backups and data archives

 * Data Dumps – August runs for English Wikipedia started on the 3rd of the month and completed on the 13th, after a couple of restarts of the history phase of the dumps.  We also cleared out the backlog of earlier incomplete dumps for May, bringing us up to current.  In the meantime, we did yet more tweaking of the filesystem to try to reduce file corruption and truncation issues.  And finally, code to "checkpoint" the history dumps by writing a sequence of smaller files was announced and tested, and it will be used for the next production run in September.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * WikiLove —
 * Feedback Dashboard —
 * GlobalProfile —
 * QuickComments —

Multimedia tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Wikimedia labs

 * TimedMediaHandler —

Mobile

 * Mobile Research — Mani Pande and Parul Vora continued to synthesize the findings from field research in India and Brazil; a research page was also created on meta. They launched user experience research in US with AnswerLab, and have started recruiting readers and editors for ethnographic research to be conducted in San Francisco, Dallas and Chicago. The mobile survey was prepared in LimeSurvey, and translations are ongoing.


 * MobileFrontend —

Fundraising support

 * 2011 Fundraiser —

Offline

 * Wikipedia version tools —
 * Kiwix UX initiative — Following the release of Kiwix 0.9 beta1 in July, Tomasz Finc invited users to test it, which surfaced issues with the Ubuntu package. A mailing list dedicated to testing Kiwix is also available.

Platform engineering
An overview of the platform engineering team was published on the Wikimedia blog in August.

MediaWiki core

 * MediaWiki 1.18 —
 * Code review management —
 * Heterogeneous deployment —
 * API maintenance —
 * Shell requests —
 * Continuous integration —
 * Wikitext scripting —
 * Projects on hold — The HipHop deployment, AcademicAccess and Disk-backed object cache projects were on hold in August.

Wikimedia analytics

 * Wikimedia Report Card 2.0 –

Technical liaison; developer relations

 * Bug management –
 * Summer of Code 2011 –
 * Engineering project documentation –
 * Volunteer coordination and outreach –
 * MediaWiki architecture document –