Wikimedia Engineering/Report/2011/July

Major news this month include:
 * Data replication from our primary Florida data center to our new Virginia data center;
 * The deployment of the Article Feedback feature to all articles on the English Wikipedia, and the deployment of MoodBar;
 * The successful implementation of a MySQL-based parser cache;
 * Mid-term evaluation of our Summer of Code projects.

Recent events

 * OSCON (July 25-29, Portland, Oregon, USA) — About a dozen Wikimedia engineers attended the Open Source Convention in late July.

Upcoming events

 * Wikimania (August 2-7, Haifa, Israel) — Another delegation of about a dozen Wikimedia engineers will be attending the Wikimania conference in early August, as well as the satellite meetings such as the Developer Days and the OpenZIM Developers Meeting. The engineering report for August will provide a more exhaustive report.


 * Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. The following positions have opened this month:
 * Product Manager (Mobile)
 * Software Developer (Mobile)
 * Product Manager (New Editor Engagement)

New Requests for Proposals:
 * Article Feedback Feature

The following positions are still open: Product Manager (Analytics), QA Lead, Operations Engineer (Networking), Director of Features Engineering), Systems Engineer (Data Analytics), Networking Contractor (Amsterdam), Software Developer (Rich Text Editing, Features), Software Developer (Front-end) and Software Developer (Back-end).

Short news

 * Jeff Green was hired as Operations Engineer for Special Projects (announcement).
 * The operations team continued to grow with Ben Hartshorne and contractor Daniel Zahn joining as Operations Engineers.
 * Ian Baker joined the Feature engineering team as Software Developer (announcement).
 * Chief technology officer Danese Cooper and Code maintenance engineer Priyanka Dhanda left the Wikimedia Foundation in July (announcements:, ).

Site infrastructure

 * Tampa Data Center — 74 new servers were purchased to increase the capacity of our Apache cluster; they will be installed in August. Network maintenance was also performed to install a new router and replace a core switch. A number of servers were upgraded, and automated with puppet.


 * Virginia Data Center — Full network connectivity was set up and the 7 wiki database clusters have now been replicated to our new servers in Virginia. We have also standardized the puppet configuration and enabled LVM snapshots. About 20 other databases (of tools like OTRS, CiviCRM, Bugzilla, WordPress and RT)  have been replicated as well. Next steps include rolling some of our Varnish caching servers, after a stability and performance assessment.


 * Media Storage — The SwiftMedia extension developed by Russ Nelson now supports all the major media features such as download, upload, re-upload, revert, delete, and restore. Upcoming work includes unit tests and performing end-to-end tests.


 * HTTPS & IPv6 — HTTPS was enabled on a private production wiki to test functionality and uncover bugs. Protocol-relative URLs (which will be a major feature of MediaWiki 1.18) were enabled on testwiki for community testing before rolling out to all projects (read more).

Testing environment

 * Virtualization test cluster —

Backups and data archives

 * Data Dumps — The June and July runs of the English Wikipedia dump were completed, and the August run is underway; possible explanations for the resolution of issues different NFS mounting options, and fine-tuning of the number of concurrent jobs. Chinese Wikipedia dumps have also been fixed. Upcoming work is focusing on checkpoint files of history dumps, to break out in-progress dumps into chunks.

Editing tools

 * Visual editor —
 * I18n/L10n tools —

Content Quality and Editorial Tools

 * Article feedback —

Participation and editor retention

 * WikiLove —
 * MoodBar —
 * GlobalProfile (formerly "StructuredProfile") —
 * LiquidThreads 3.0 —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Wikimedia Labs

 * TimedMediaHandler —


 * New parser — Brion Vibber continued to work on the ParserPlayground extension, which is now a mostly working demo. He's now focusing on the API between the parser/renderer and its host environment (read more).

Mobile

 * Mobile Research —


 * MobileFrontend —

Fundraising support

 * 2011 Fundraiser —

Offline

 * Wikipedia version tools — GSoC student Yuvaraj Pandian continued to port the WP 1.0 bot to a MediaWiki extension. Mentored by Arthur Richards, and supported by WP 1.0 Bot author/maintainer User:CBM, Yuvaraj nearly achieved feature parity with the WP 1.0 bot by implementing article selection filtering based on project, quality, importance and category.  He also implemented the ability to save lists of filtered articles.  In the coming month, Yuvaraj will wrap up the initial development by adding the ability to manually curate article selection lists and export article lists in CSV format.


 * Collections —


 * Kiwix UX initiative —

We've been hard at work on our next set of beta releases for Kiwix and are proud to announce the release of beta 0.9 rc1. With this new version were introducing a new content manager, refine search results, and fixes from our first usability study. For those tracking our changes head over to the changelog. We spent a lot of time refining our build system so that the next set of beta can come out ever faster. Stay tuned for some blog posts about how to get involved by testing.

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Heterogeneous deployment —
 * Disk-backed object cache —
 * API maintenance — Sam Reed continued to fix bugs and to add new features to the MediaWiki API.  Sam's API work in July focused on providing the API component to the new Report Card project.
 * Shell requests — Sam Reed took over maintenance of shell bugs.  He added a new "ops" keyword to differentiate between bugs that require the shell access that he has, and other requests that require root access ("ops") that requires someone who has root access.  As of July 26, we were down to 69 shell requests (and that number has gone down since then).
 * Continuous integration —
 * Projects on hold — The HipHop deployment, AcademicAccess, App-level monitoring and Configuration management projects were mostly on hold in July.

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * Bug management —
 * Summer of Code 2011 —
 * (read more)


 * Engineering project documentation —
 * Volunteer coordination and outreach —