Wikimedia Engineering/Report/2011/December

Major news in December include:
 * The deployment of WebFonts to select Indic wikis;
 * The first developer prototype of the Visual editor;
 * A new version of Article Feedback being tested on the English Wikipedia;
 * Progress on the Swift media storage project.

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * Judging for the October 2011 Coding Challenge continued and winners will be announced in January.

Upcoming events

 * San Francisco hackathon (21–22 January 2012) — Erik Möller and Sumana Harihareswara continued to plan and publicize this outreach-focused developers' week-end. Heather Walls developed a more attractive homepage for the event. Sumana began arranging for tutorials and activities for the event, focusing on mobile, the web-accessible API and our framework for JavaScript feature development. Registration opened and more than 70 participants registered.


 * Pune hackathon (10-12 February 2012) — Preparation began and registration opened for an outreach-focused developers' week-end to take place in Pune, India, and led by Alolita Sharma. Approximately 70 participants are expected, focusing on the gadgets framework, mobile Wikimedia access, and internationalisation.


 * GLAMCamp in Washington, DC (10-12 February 2012) — Ryan Kaldari and Asaf Bartov plan to attend the technical track of this GLAM conference. Engineers will work on mass upload and analytics functionality.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Interaction Designer
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer


 * Management & Product:
 * Director of Features Engineering
 * Product Manager


 * Requests for proposals:
 * Executive Dashboard - Analytics — Help us improve and centralize the dashboard summarizing the most important data for both Wikimedia Foundation staff and projects such as Wikipedia to understand overall community health.
 * XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Yuvaraj Pandian and Max Semenik joined the mobile team as contract developers.
 * Sara Smollett joined the operations team as a part-time contractor.
 * Diederik van Liere, formerly with the Community Department, is now helping the engineering department as a contractor for analytics work.

Site infrastructure

 * Data Centers — The team deployed a new MediaWiki profiling system based on graphite, to track performance across the application stack, and to provide statistics/graphing as a service for MediaWiki within the WMF production environment.  Some database servers were moved to newer hardware including OTRS, and those in the Ashburn data center were upgraded to a new build of mysql-at-facebook. Mark refactored our configuration tool (Puppet) to address scalability and performance issues.


 * Media Storage — As part of our preparation for the migration of our media service to Swift, a distributed storage back-end, we need to keep the current system afloat a bit longer. We reclaimed some space by purging thumbnails not newly generated and not in use on any of our projects. We also performed Swift thumbnail integration and stress testing. Read performance is about 10x what we need on the performance test cluster so we're good on that front. Write performance is only 2x what we need, but sufficient to move forward. Tests and research indicates performance drops over a few million objects; the easiest path forward is to shard the Commons container using the existing hashed characters in the URL, splitting the container into 256 containers.


 * HTTPS — HTTPS support was added for mobile, for Wikipedia. After an initial testing period, we'll enable this for further mobile sites. A number of other miscellaneous services also had HTTPS set up or fixed.

Testing environment

 * Wikimedia Labs — A server admin log was created for every project, as well as a [//labsconsole.wikimedia.org/wiki/Server_Admin_Log combined log]. OpenStackManager 1.3 and LdapAuthentication 2.0a were deployed to Labs. Live migration of instances has been enabled for the OpenStack Nova infrastructure, allowing updates and upgrade of hardware without bringing instances down. A gluster storage cluster has been ordered for use as volume storage. A number of projects were added or moved to Labs, including adminbot, nagios, Cluebot, testswarm and the reportcard service. There are now 33 projects, 52 instances, and 74 users.

Backups and data archives

 * Data Dumps — The end of the year closed out with another full dump of the English language Wikipedia on schedule.  More work was done on code to allow restart of the history phase of a dump from a specified point without a long catchup delay.  An experimental service was tested this month: a newly formatted file of article content and an accompanying index, more convenient for data analysts and for use with offline readers.  The first such files, only available for the English Wikipedia, are available here along with a brief explanation of their contents here.

Editing tools

 * Visual editor —
 * Internationalization and localization tools —

Participation and editor retention

 * Article feedback —
 * Feedback Dashboard —

Multimedia Tools

 * UploadWizard —
 * TimedMediaHandler —

MediaWiki infrastructure

 * ResourceLoader —

Feature support

 * Wikipedia Education Program —

Mobile

 * Mobile Research — Mani Pande and Parul Vora consolidated all the research findings from Brazil, India, and the USA into one report. It's currently being converted to PDF and wikitext to facilitate its publication.


 * MobileFrontend —


 * Android Wikipedia App — Several release candidates were released over the month and we're nearing completion of the first version of the app. Both Yuvi and Brion have been developing furiously and need more testers to get involved. Nightly builds are available for testing.


 * WikipediaZero — We began work on the infrastructure for zero rated Wikipedia access. Next month, we'll start testing with one of our partners to work out the kinks of giving users free data access to Wikipedia.


 * GPS Storage/Retrieval — Max Semenik joined the mobile team and began prototying an API to store and retrieve GPS coordinates on our wikis. This will be a critical component of the mobile projects and will replace our existing use of GeoNames.org and can also supplement GeoHack.


 * Featured Article RSS — Max Semenik built the first version of an extension to expose Featured Articles, In the news, and other main page content so that our partners can better re-use our data.

Fundraising support

 * 2011 Fundraiser —

Offline

 * Kiwix UX initiative

Work continued on the 0.9 release of Kiwix. Lead developer Emmanuel Engelhart released beta5 for community testing fixing lots of reported bugs and soliciting testers to get involved.

MediaWiki Core

 * The "MediaWiki Core" team was featured on the Wikimedia Tech blog this month.


 * MediaWiki 1.19 —
 * Continuous integration —
 * Git conversion —
 * VipsScaler —
 * SwiftMedia —
 * HipHop deployment —

Wikimedia analytics

 * Wikimedia Report Card 2.0 —

Technical Liaison; Developer Relations

 * Bug management —
 * Summer of Code 2011 —
 * Engineering project documentation —
 * Volunteer coordination and outreach —
 * MediaWiki architecture document —
 * Wikimedia blog maintenance —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.