Wikimedia Engineering/Report/2011/July

Major news in July include:
 * Ongoing data replication from our primary Florida data center to our new Virginia data center;
 * The deployment of the Article Feedback feature to all articles on the English Wikipedia, and the deployment of MoodBar;
 * The successful implementation of a MySQL-based parser cache on Wikimedia wikis;
 * Mid-term evaluation of our Summer of Code projects.

Recent events

 * OSCON (July 25-29, Portland, Oregon, USA) — About a dozen Wikimedia engineers attended the Open Source Convention in late July. OSCON is used to showcase the latest and greatest developments in open source technologies (including hands-on tutorials), and is generally an opportunity for Wikimedia developers to stay in the loop and to network with individuals from other projects and communities. We had two presentations in the program (on the 2010-11 fundraising campaign, and on ResourceLoader), which are available in the Wikimedia engineering presentations collection. We also promoted WMF job openings at every opportunity. Finally, Danese Cooper, Sumana Harihareswara and Erik Moeller participated in a workshop with like-minded organizations regarding volunteer matching strategies for open source projects.

Upcoming events

 * Wikimania (August 2-7, Haifa, Israel) — Another delegation of about a dozen Wikimedia engineers will be attending the Wikimania conference in early August, as well as the satellite meetings such as the Developer Days and the OpenZIM Developers Meeting. The engineering report for August will provide a more exhaustive report.


 * Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. The following positions opened in July:
 * Product Manager (Mobile)
 * Software Developer (Mobile)
 * Product Manager (New Editor Engagement)

New Requests for Proposals:
 * Article Feedback Feature

The following positions are still open: Product Manager (Analytics), QA Lead, Operations Engineer (Networking), Director of Features Engineering, Systems Engineer (Data Analytics), Networking Contractor (Amsterdam), Software Developer (Rich Text Editing, Features), Software Developer (Front-end) and Software Developer (Back-end).

Short news

 * Jeff Green was hired as Operations Engineer for Special Projects (announcement).
 * The operations team continued to grow with Ben Hartshorne and contractor Daniel Zahn joining as Operations Engineers (announcement).
 * Ian Baker joined the Feature engineering team as Software Developer (announcement).
 * Chief technology officer Danese Cooper and Code maintenance engineer Priyanka Dhanda left the Wikimedia Foundation in July (announcements:, ).

Site infrastructure

 * Tampa Data Center — 74 new servers were purchased to increase the capacity of our Apache cluster; they will be installed in August. Network maintenance was also performed to install a new router and replace a core switch. A number of servers were upgraded, and automated with puppet.


 * Virginia Data Center — Full network connectivity was set up and the 7 wiki database clusters have now been replicated to our new servers in Virginia. We have also standardized the puppet configuration and enabled LVM snapshots. About 20 other databases (of tools like OTRS, CiviCRM, Bugzilla, WordPress and RT)  have been replicated as well. Next steps include rolling out some of our Varnish caching servers, after a stability and performance assessment.


 * Media Storage — The SwiftMedia extension developed by Russ Nelson now supports all the major media features such as download, upload, re-upload, revert, delete, and restore. Upcoming work includes unit tests and performing end-to-end tests.


 * HTTPS & IPv6 — HTTPS was enabled on a private production wiki and testwiki to test functionality and uncover bugs. Protocol-relative URLs (which will be a major feature of MediaWiki 1.18) were enabled on testwiki for community testing before rolling out to all projects (read more).

Testing environment

 * Virtualization test cluster — This project was slowed down in favor of deploying HTTPS. Some work was done to move the puppet configuration into a public repository.

Backups and data archives

 * Data Dumps — The June and July runs of the English Wikipedia dump were completed, and the August run is underway; possible explanations for the resolution of issues include different NFS mounting options, and fine-tuning the number of concurrent jobs. Chinese Wikipedia dumps have also been fixed. Upcoming work is focusing on checkpoint files of history dumps, to break out in-progress dumps into chunks.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Content Quality and Editorial Tools

 * Article feedback —

Participation and editor retention

 * WikiLove —
 * MoodBar —
 * GlobalProfile (formerly "StructuredProfile") —
 * LiquidThreads 3.0 —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Wikimedia Labs

 * TimedMediaHandler —
 * MediaWiki.next — Brion Vibber continued to work on the ParserPlayground extension, which is now a mostly working demo. He's now focusing on the API between the parser/renderer and its host environment (read more).

Mobile

 * Mobile Research — Parul Vora and Mani Pande continued to plan the US mobile research, to talk to possible firms, and to draft the mobile survey. Reports and syntheses from the India and Brazil field research were delayed in favor of the US research planning.


 * MobileFrontend —

Fundraising support

 * 2011 Fundraiser —

Offline

 * Wikipedia version tools —
 * Kiwix UX initiative — Kiwix 0.9 beta1 was released in July and included a new content manager, better search results, and fixes from our first usability study (more details in the changelog). We also refined our build system to speed up the release process.

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Heterogeneous deployment —
 * Disk-backed object cache —
 * API maintenance —
 * Shell requests —
 * Continuous integration —
 * Projects on hold — The HipHop deployment, AcademicAccess, App-level monitoring and Configuration management projects were mostly on hold in July.

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —