Wikimedia Engineering/Report/2011/November

Major news in November include:

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * October 2011 Coding Challenge (20 October - 7 November, online) — The coding challenge was successfully wrapped up; the submissions have been published and will be judged in December.


 * India hackathon (18–20 November, Mumbai, India) — Approximately 80 Indian participants came to this week-end hackathon, focusing on Wikimedia mobile, offline, and language issues. According to Phil Chang, participants made great strides in mobile. The internationalization effort also benefited, with new input methods for MediaWiki, readying Narayam for Wikimedia Incubator, a prototype onscreen keyboard built in Narayam, Wikimedia Mobile ready for translation, new UI prototypes for language selection, and other new features. Participants localized Kiwix into several Indic languages. Volunteer Development Coordinator Sumana Harihareswara summarized the event on wikitech-l.


 * Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte and Tom Morris organized a general MediaWiki hackathon with approximately 10 participants. Attendees fixed bugs, reviewed outstanding patches from volunteers, enhanced OpenPlaques with the MediaWiki API, and discussed Wikinews and Wikimedia Commons.

Upcoming events

 * San Francisco hackathon (21-22 January 2012) — Erik Möller and Sumana Harihareswara continued to plan and started publicizing an outreach-focused developers' week-end. Experienced staff and volunteer developers will participate, "teaching you about MediaWiki (our software), about our API and about our framework for JavaScript feature development. Learn to reuse Wikipedia content in your own apps, create new functionality for every Wikipedia user around the world, or just tailor your own experience."

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer.


 * Management & Product:
 * Director of Features Engineering
 * Product Manager
 * Product Manager (Analytics)


 * Requests for proposals:
 * API Logging Analysis — Help us analyze the query logs of our API to better understand third party application usage of Wikimedia content and services.
 * XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Benny Situ and Rob Moen joined the Features engineering team as Software Developers, to work on Editor Engagement features (announcement for Benny, announcement for Rob).
 * Andrew Bogott has joined the Operations team as a developer to work on the Labs Virtualization project
 * Aaron Halfaker, former Summer of research fellow, joined the Features team as a research contractor for data analysis on editor engagement features.

Site infrastructure

 * Data Centers — Eight new databases servers for external storage were deployed in Ashburn and Tampa, to retire about 30 aging servers. The goal was to add capacity, improve performance, consume less power and provide cross data-center data recovery and redundancy (SDTPA & EQIAD). This gave us back one rack of server space, which is significant given the space constraint at SDTPA. Also in EQIAD, servers for bits.wikimedia.org and Fundraising reporting were migrated and are now in production. In Amsderdam, SSDs were added to the ESAMS Squid servers to improve read performance, and we upgraded the ESAMS core switch (csw2-esams). The KNAMS router was moved because we changed to a new hosting provider, though still within the same datacenter. In Tampa, 66 servers were retired, donated and shipped to various non-profits. More generally, our configuration management tool Puppet was upgraded and deployed to all of our lab and production servers; a Puppet dashboard was also implemented.
 * New MySQL Package —  A new Debian package for mysqlatfacebook 5.1.53 was successfully built, using the fresh version of the facebook patchset. Initial production data on two hosts seem to be promising.


 * HTTPS — The HTTPS Everywhere project released version 1.2 of its Firefox plugin in October. Its rulesets did not properly account for the HTTPS configurations of certain Wikimedia sites, so Roan Kattouw followed up with improved rulesets in November.


 * Site outage — There was a short outage on November 27th, caused by a combination of a surge in traffic and cache stampede.

Testing environment

 * Virtualization test cluster
 * Upgraded OpenStack Nova from cactus to diablo
 * Added GlusterFS filesystem on all compute nodes via puppet, to act as storage for the instances
 * Added default sudo policy for instances. Project members now have sudo permissions, excluding global projects
 * Added shared home directories, in a per project manner
 * 15 projects and 36 instances have been created and 46 people have been given Labs accounts so far
 * The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production

Backups and data archives

 * Data Dumps — While the dumps keep rolling along, we are talking with another organization interested in mirroring them. If you know someone with several terabytes of space who might be interested, please send them our way. All hosts had their kernel updated for security reasons, and new dump code was deployed as well. We rolled out a new experimental service this month of daily adds/changes dumps for all projects. No information about deleted/undeleted/moved pages from previous dumps is included, but it does include all new content since the run of the previous day. The first adds/changes run for the English language Wikipedia took less than 30 minutes to build.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * Feedback Dashboard —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Media Labs

 * TimedMediaHandler —

Mobile

 * Mobile Research — Mani Pande and Parul Vora presented the findings from India and Brazil report to the mobile team. They are currently working on consolidating the report, videos, photographs and other media into a Wiki format. The cleaning of data from the mobile survey continues.


 * MobileFrontend —


 * Android Wikipedia App — Were tidying up the app now and would love to get testers. Send a mail to the mobile-l-feedback list if you want to be involved.

Fundraising support

 * 2011 Fundraiser —

Offline

 * Kiwix UX initiative —

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Shell requests —
 * Continuous integration —
 * Git conversion —
 * VipsScaler —
 * SwiftMedia —

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * The "Technical Liaison; Developer Relations" team was featured on the Wikimedia Tech blog this month.


 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —
 * MediaWiki architecture document —
 * Wikimedia blog maintenance —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.