Wikimedia Engineering/Report/2011/November

Major news in November include:
 * The completion of the Coding challenge, and two coding events in India and the UK;
 * Continued infrastructure work in our data centers to improve performance and reliability, and on the Labs project;
 * Progress on the Visual editor and its back-end;
 * New versions of the Feedback Dashboard and the Upload Wizard, bringing new and long-awaited features;
 * Fundraising engineering going full-swing, in parallel with the annual fundraising campaign;
 * The final release of MediaWiki 1.18.0.

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * October 2011 Coding Challenge (20 October - 7 November, online) — The coding challenge was successfully wrapped up; the submissions have been published and will be judged in December.


 * India hackathon (18–20 November, Mumbai, India) — Approximately 80 Indian participants came to this week-end hackathon, focusing on Wikimedia mobile, offline, and language issues. According to Phil Chang, participants made great strides in mobile. The internationalization effort also benefited, with new input methods for MediaWiki, readying Narayam for Wikimedia Incubator, a prototype onscreen keyboard built in Narayam, Wikimedia Mobile ready for translation, new UI prototypes for language selection, and other new features. Participants localized Kiwix into several Indic languages. Volunteer Development Coordinator Sumana Harihareswara summarized the event on wikitech-l.


 * Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte and Tom Morris organized a general MediaWiki hackathon with approximately 10 participants. Attendees fixed bugs, reviewed outstanding patches from volunteers, enhanced OpenPlaques with the MediaWiki API, and discussed Wikinews and Wikimedia Commons.

Upcoming events

 * San Francisco hackathon (21-22 January 2012) — Erik Möller and Sumana Harihareswara continued to plan and started publicizing an outreach-focused developers' week-end. Experienced staff and volunteer developers will participate, "teaching you about MediaWiki (our software), about our API and about our framework for JavaScript feature development. Learn to reuse Wikipedia content in your own apps, create new functionality for every Wikipedia user around the world, or just tailor your own experience."

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer.


 * Management & Product:
 * Director of Features Engineering
 * Product Manager
 * Product Manager (Analytics)


 * Requests for proposals:
 * API Logging Analysis — Help us analyze the query logs of our API to better understand third party application usage of Wikimedia content and services.
 * XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Benny Situ and Rob Moen joined the Features engineering team as Software Developers, to work on Editor Engagement features (announcement for Benny, announcement for Rob).
 * Andrew Bogott has joined the Operations team as a developer to work on the Labs Virtualization project
 * Aaron Halfaker, former Summer of research fellow, joined the Features team as a research contractor for data analysis on editor engagement features.

Site infrastructure

 * Data Centers — Eight new database servers for external storage were deployed in Ashburn and Tampa, to retire about 30 aging servers. The goal was to add capacity, improve performance, consume less power and provide cross data-center data recovery and redundancy (SDTPA & EQIAD). This gave us back one rack of server space, which is significant given the space constraint at SDTPA. Also in EQIAD, servers for bits.wikimedia.org and Fundraising reporting were migrated and are now in production. In Amsterdam, SSDs were added to the ESAMS Squid servers to improve read performance, and we upgraded the ESAMS core switch (csw2-esams). The KNAMS router was moved because we changed to a new hosting provider, though still within the same datacenter. In Tampa, 66 servers were retired, donated and shipped to various non-profits. More generally, our configuration management tool Puppet was upgraded and deployed to all of our lab and production servers; a Puppet dashboard was also implemented.
 * New MySQL Package —  A new Debian package for mysqlatfacebook 5.1.53 was successfully built, using the fresh version of the facebook patchset. Initial production data on two hosts seem to be promising.


 * HTTPS — The HTTPS Everywhere project released version 1.2 of its Firefox plugin in November. This version includes the updated Wikimedia ruleset written by Roan Kattouw, which redirects users to the new HTTPS URLs rather than to the old secure gateway.


 * Site outage — There was a short outage on November 27th, caused by a combination of a surge in traffic and cache stampede.

Testing environment

 * Virtualization test cluster — OpenStack Nova was upgraded from cactus to diablo. A GlusterFS filesystem was added on all compute nodes via puppet, to act as storage for the instances. A default sudo policy was also added for instances: project members now have sudo permissions, excluding global projects. Shared home directories are also available in a per project manner. 15 projects and 36 instances have been created and 46 people have been given Labs accounts so far. The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production.

Backups and data archives

 * Data Dumps — While the dumps keep rolling along, we are talking with another organization interested in mirroring them. If you know someone with several terabytes of space who might be interested, please send them our way. All hosts had their kernel updated for security reasons, and new dump code was deployed as well. We rolled out a new experimental service this month of daily adds/changes dumps for all projects. No information about deleted/undeleted/moved pages from previous dumps is included, but it does include all new content since the run of the previous day. The first adds/changes run for the English language Wikipedia took less than 30 minutes to build.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * Feedback Dashboard —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Media Labs

 * TimedMediaHandler —

Mobile

 * Mobile Research — Mani Pande and Parul Vora presented the findings from India and Brazil report to the mobile team. They are currently working on consolidating the report, videos, photographs and other media into a Wiki format. The cleaning of data from the mobile survey continues.


 * MobileFrontend —


 * Android Wikipedia App — We're tidying up the app now and would love to get testers. Send a mail to the mobile-l-feedback list if you want to be involved.

Fundraising support

 * 2011 Fundraiser —

Offline

 * Kiwix UX initiative —

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Continuous integration —
 * Git conversion —
 * VipsScaler —
 * SwiftMedia —

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * The "Technical Liaison; Developer Relations" team was featured on the Wikimedia Tech blog this month.


 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —
 * MediaWiki architecture document —
 * Wikimedia blog maintenance —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.