Wikimedia Engineering/Report/2011/November

Major news in November include:

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * October 2011 Coding Challenge (20 October - 7 November, online) — The coding challenge was successfully wrapped up, and the submissions have been published and will be judged in December.


 * India hackathon (18–20 November, Mumbai, India) — Approximately 80 Indian participants came to this week-end hackathon, focusing on Wikimedia mobile, offline, and language issues. According to Phil Chang, participants made great strides in mobile. The internationalization effort also benefited, with new input methods for MediaWiki, readying Narayam for Wikimedia Incubator, a prototype onscreen keyboard built in Narayam, Wikimedia Mobile ready for translation, new UI prototypes for language selection, and other new features. Participants localized Kiwix into several Indic languages. Volunteer Development Coordinator Sumana Harihareswara summarized the event on wikitech-l.


 * Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte and Tom Morris organized a general MediaWiki hackathon with approximately 10 participants. Attendees fixed bugs, reviewed outstanding patches from volunteers, enhanced OpenPlaques with the MediaWiki API, and discussed Wikinews and Wikimedia Commons.

Upcoming events

 * San Francisco hackathon (21-22 January 2012) — Erik Möller and Sumana Harihareswara continued to plan and started publicizing an outreach-focused developers' week-end. Experienced staff and volunteer developers will participate, "teaching you about MediaWiki (our software), about our API and about our framework for JavaScript feature development. Learn to reuse Wikipedia content in your own apps, create new functionality for every Wikipedia user around the world, or just tailor your own experience."

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer.


 * Management & Product:
 * Director of Features Engineering
 * Product Manager
 * Product Manager (Analytics)


 * Requests for proposals:
 * API Logging Analysis — Help us analyze the query logs of our API to better understand third party application usage of Wikimedia content and services.
 * XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Benny Situ and Rob Moen joined the Features engineering team as Software Developers, to work on Editor Engagement features (announcement for Benny, announcement for Rob).
 * Andrew Bogott has joined us as a TechOps developer to work on the Labs Virtualization project
 * Aaron Halfaker, former summer of research fellow, joined the feature team as a research contractor for data analysis on editor engagement features.

Site infrastructure

 * Data Centers —
 * External Store : 8 New database (aka External Store) servers were deployed to retire about 30 aging servers in order to add capacity, improve performance, consume less power and provide cross data-center data recovery and redundancy (SDTPA & EQIAD). This allows us to recover 1 rack of server space, which is especially significant given the space constraint at SDTPA.
 * ESAM Squid Server upgrade : SSDs were added to the ESAM Squid servers to improve read performance.
 * ESAM Core Switch: Upgraded csw2-esams
 * KNAMS router move: Router had to be moved because we had to change to a new hosting provider, though still within the same datacenter.
 * Retired servers : Donated & shipped 66 servers from Tampa were to various non-profits
 * EQIAD deployment : bits.wikipedia.org together with Fundraising reporting servers are now migrated and in production at our new datacenter
 * Puppet (for Configuration Management) : Upgraded (upgraded to 2.7.7rc2 from 2.6) and deployed to all of our lab and production servers; Mark also implemented puppet dashboard.
 * Others: There was a short outage (http://wikitech/view/Site_issue_November_27,_2011) on the 27th caused by a combination of a surged in traffic and cache stampede ( http://en.wikipedia.org/wiki/Cache_stampede)


 * Media Storage — Ben Hartshorne has set up the initial development environment for Engineering to have a platform on which to test and continue development of the SwiftMedia extension. Ben and Mark Bergsma continue to do performance testing on Swift prior to using it in production.  Aaron Schulz started to refactor the FileBackend extension, a requirement to using SwiftMedia.
 * New MySQL Package - clean shiny debian packaging of mysqlatfacebook 5.1.53 . Finally done using the very fresh version of the facebook patchset, built with compiler and config options Domas would approve of. So far it seems to be working great.   Binaries are in the wikimedia-lucid repo and it's currently running in production on two hosts.


 * HTTPS — The HTTPS Everywhere project released version 1.2 of its Firefox plugin in October. Its rulesets did not properly account for the HTTPS configurations of certain Wikimedia sites, so Roan Kattouw followed up with improved rulesets in November.

Testing environment

 * Virtualization test cluster
 * Upgraded OpenStack Nova from cactus to diablo
 * Added GlusterFS filesystem on all compute nodes via puppet, to act as storage for the instances
 * Added default sudo policy for instances. Project members now have sudo permissions, excluding global projects
 * Added shared home directories, in a per project manner
 * 15 projects and 36 instances have been created and 46 people have been given Labs accounts so far
 * The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production

Backups and data archives

 * Data Dumps — While the dumps keep rolling along, we are talking with another organization interested in mirroring them. If you know someone with several terabytes of space who might be interested, please send them our way. All hosts had their kernel updated for security reasons, and new dump code was deployed as well. We rolled out a new experimental service this month of daily adds/changes dumps for all projects. No information about deleted/undeleted/moved pages from previous dumps is included, but it does include all new content since the run of the previous day. The first adds/changes run for the English language Wikipedia took less than 30 minutes to build.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * Feedback Dashboard —

Multimedia Tools

 * UploadWizard —

MediaWiki infrastructure

 * ResourceLoader —

Media Labs

 * TimedMediaHandler —

Mobile

 * Mobile Research — Mani Pande and Parul Vora presented the findings from India and Brazil report to the mobile team. They are currently working on consolidating the report, videos, photographs and other media into a Wiki format. The cleaning of data from the mobile survey continues.


 * MobileFrontend —


 * Android Wikipedia App — Were tidying up the app now and would love to get testers. Send a mail to the mobile-l-feedback list if you want to be involved.

Fundraising support

 * 2011 Fundraiser —

Offline

 * Kiwix UX initiative —

MediaWiki Core

 * MediaWiki 1.18 —
 * Code review management —
 * Shell requests —
 * Continuous integration —
 * Git conversion —
 * VipsScaler — Tim Starling and Antoine Musso reviewed the VipsScaler extension, written by Bryan Tong Minh to use VIPS as an alternative, and possible replacement, for ImageMagick as image thumbnailing system. Bryan and Antoine also wrote a comparison tool, [//test2.wikipedia.org/wiki/Special:VipsTest enabled on the test2 wiki], to test both systems. The initial deployment is planned to be limited to images that would give an error with ImageMagick (for instance large PNG files).
 * SwiftMedia -

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * The "Technical Liaison; Developer Relations" team was featured on the Wikimedia Tech blog this month.


 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —
 * MediaWiki architecture document —
 * Wikimedia blog maintenance —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.