Wikimedia Engineering/Report/2011/October

Major news in October include:
 * The New Orleans hackathon, which focused on Wikimedia's infrastructure;
 * Progress on the Visual editor project, with the first prototype expected in the coming months;
 * The deployment of the Translate extension to meta-wiki;
 * The deployment of MediaWiki 1.18 to all Wikimedia sites;
 * The completion of the first revision of the MediaWiki architecture document;
 * The ramp-up by the fundraising engineering team, to prepare for the upcoming annual fundraiser.

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * New Orleans hackathon (14–16 October, New Orleans, USA) — About 30 MediaWiki developers and friends attended this coding event (organized by Ryan Lane, Sumana Harihareswara and Dana Isokawa) to work on Wikimedia's technical infrastructure. Work focused on the virtualization cluster / Wikimedia Labs, media storage (using Swift), Puppet, and continuous integration. The event was also used to discuss and prepare for the upcoming migration to git as our primary version control system.


 * October 2011 Coding Challenge (20 October - 7 November, online) — Greg DeKoenigsberg continued to prepare for this contest, which officially started on October 20th, and will be running for a few more days (see the rules). The contest offers three challenges: uploading media via a smartphone, surfacing changes in real time, and showing slideshows. The back-end, which provides a sign-up and submission process, was developed by Jeroen De Dauw and released as a MediaWiki extension. You can still participate in the challenge.

Upcoming events

 * India hackathon (18–20 November, Mumbai, India) — Alolita Sharma, Siebrand Mazeland, Sumana Harihareswara and the local India team continued to prepare for this event, which will focus on language, mobile and offline support for MediaWiki content. You can register to request a free invitation; approximately 100-125 attendees are expected.


 * Brighton hackathon (19–20 November, Brighton, England) — Free registration opened for this general MediaWiki hackathon planned by Lewis Cawte.  You can register online.  WMF engineers Antoine Musso, Roan Kattouw, and Sam Reed plan to attend.


 * San Francisco event (21-22 January 2012) — Erik Moeller and Sumana Harihareswara worked on planning for an outreach-focused developers' weekend. Staff and volunteer developers will help interested technologists learn to make awesome things with Wikimedia, by improving the Wikimedia platform, creating new functionality for Wikimedia users, or integrating Wikimedia content into their own products and services.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

One new position opened in October: Software Security Engineer, and an RfP was published: RFP/Mobile UI/UX Redesign.

The following positions are still open:
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Director of Features Engineering
 * Software Developer (Mobile)
 * Product Manager (Analytics)
 * Product Manager.

The following requests for proposals are still open:
 * Logging Analysis
 * Development and Operations Engineer
 * XML Dumps Help.

Short news

 * Phil Chang joined the Mobile team as Product Manager (announcement).
 * Leslie Carr joined the Operations Team as Operations and Network Engineer (announcement).
 * Antoine Musso joined the MediaWiki Core team as a contractor working on Continuous integration (announcement).
 * Amir Aharoni joined the Internationalization and localization tools team as Software developer (announcement).
 * Two contractors joined the Strategic Product team: Fabrice Florin, leading the Article feedback 5.0 feature, and Oliver Keyes, acting as Community liaison (announcement).
 * Gabriel Wicke joined the Features engineering team as Software developer, focusing on the Visual editor (announcement).
 * Yoni Shostak (project manager), Greg Chiasson (developer), Reha Sterbin (developer) and Sean Heavey (UI designer/front-end developer), from OmniTI, will be working on the Article Feedback 5.0 feature.

Site infrastructure

 * Data Centers — Chris Johnson, Rob Halsell and Peter Youngmeister have racked and brought online new servers in Tampa to enhance the capacity of the Bits, MediaWiki and API application servers. Leslie Carr worked on partitioning with PartMan and documented the process. An external firm performed a security audit on our infrastructure on our behalf. The team also addressed several high priority break fixes, like the external data storage failures, and the analytics packet loss due to heavy load with udp2log processing. Peter Youngmeister upgraded all Apache servers to Ubuntu 10.04; Squid servers have also been upgraded, and the Linux kernels have been updated with the latest security patches. Ben Hartshorne continued to set up servers for external storage. Mark Bergsma and Asher Feldman deployed two new servers running a new and improved version of the Bits caching server at our ESAM infrastructure, in preparation to deploying it at Eqiad.


 * Media Storage — During the New Orleans hackathon, developers agreed to create an abstraction layer (FileBackend) which will allow MediaWiki users to use SwiftMedia to store media files, or another filesystem of their choice. They also agreed to re-architect the FileRepo module in Mediawiki.


 * HTTPS — HTTPS support has been enabled on all Wikimedia sites; mobile support is still to come. At some point in the future, we'll be switching the log-in link to default to logging in via https.

Testing environment

 * Virtualization test cluster — Wikimedia Labs has been launched. We are starting to install instances to duplicate our production environment. We are also bringing up some instances for non-MediaWiki related work. At the NOLA hackathon, we worked with a team from Canonical to give the juju project access to our environment. A new project from Wikimedia Deutschland, Wikidata, announced they'd be working in the Labs environment as well.

Backups and data archives

 * Data Dumps — This month's dump for the English Wikipedia started later in the month (on October 11), since we were waiting for the production roll-out of MediaWiki 1.18. The run completed without issues, and all other servers are also using 1.18 without problems. We also have our first mirror site up and running, hosted by the C3SL at Universidade Federal do Paraná in Brazil, whom we thank for their support. The mirror provides a copy of the last 5 successful dumps of each project and is updated as new dumps complete.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * LiquidThreads 3.0 —
 * Feedback Dashboard —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Media Labs

 * TimedMediaHandler —

Mobile

 * Mobile Research — Mani Pande and Parul Vora conducted user experience research in the US in three cities (San Francisco, Chicago and Dallas) assisted by UX firm AnswerLab. They met with mobile readers, potential mobile readers and editors. They are also finalizing the report on their field research in India and Brazil.


 * MobileFrontend —


 * Android Wikipedia App — We partnered with nitobi to create the first official Wikipedia app for Android devices. The app is based on the PhoneGap framework and should be added to the Android market soon. The team is still looking for developers.

Fundraising support

 * 2011 Fundraiser —

Offline

 * Kiwix UX initiative — The last critical bugs in version 0.9 of Kiwix (especially on Windows) were fixed, and the release candidate cycle will soon begin. A Kiwix activity was released in alpha for Sugar, a learning platform for children. The first two prototypes of the Kiwix-plug have been built; the Kiwix-plug is a standalone wifi hotspot providing offline Wikipedia content, based on a plug computer.

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Shell requests —
 * Continuous integration —
 * Git conversion —

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —
 * MediaWiki architecture document —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.