Wikimedia Engineering/Report/2012/January

Major news in January include:
 * Tech support for the black-out protesting SOPA & PIPA;
 * The release of the official Wikipedia Android app;
 * A new beta cluster for Wikimedians to test upcoming software before it's deployed to production.

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * English Wikipedia anti-SOPA blackout — The engineering team supported this online protest by developing and deploying the blackout code and design, including the CongressLookup extension for helping people  find and contact their representative, pulling from Sunlight Foundation APIs and other sources. Operations disabled editing during the 24 hour time period, and helped keep other systems up and running, including the temporarily overloaded Wikimedia blog.


 * San Francisco hackathon (21–22 January 2012, San Francisco, California, USA) — More than 90 participants learned and hacked during this outreach-focused developers week-end. Sumana Harihareswara and several other WMF staffers worked with the coworking venue pariSoma to organize the event.  The teams of participants demonstrated more than a dozen projects.  The demos, a speech by Erik Möller, and tutorials in mobile, Gadgets, and the web API were recorded, and recordings are forthcoming.


 * October 2011 Coding Challenge — The winners of the coding challenge were announced. They include an Android app for uploading to Wikimedia Commons, a user script for surfacing pages with a lot of recent editing activity, and a user script for displaying relevant images in an article as a lightbox slideshow.

Upcoming events

 * Pune hackathon (10–12 February 2012, Pune, India) — Preparation began and registration continued for an outreach-focused developers week-end to take place in Pune, India, and led by Alolita Sharma. Approximately 70 participants are expected to learn and contribute at this event, focusing on the gadgets framework, mobile Wikimedia access, and internationalization.


 * GLAMcamp DC (10–12 February 2012, Washington, D.C., USA) — The Foundation's Ryan Kaldari and Asaf Bartov plan to attend the technical track of this GLAM conference. Engineers will work on mass upload and analytics functionality, which cultural institutions find useful in partnering with Wikimedia.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Senior Software Engineer Front-end
 * Interaction Designer
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer
 * Operations Engineer (Labs)


 * Management & Product:
 * Director of Features Engineering


 * Requests for proposals:
 * Executive Dashboard - Analytics — Help us improve and centralize the dashboard summarizing the most important data about the Wikimedia movement to understand overall community health.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Andrew Otto joined the Platform engineering team as Software Developer for Analytics ([//lists.wikimedia.org/pipermail/wikitech-l/2012-January/057430.html announcement]).
 * Fabrice Florin joined the Strategic Product team as Product Manager for New Editor Engagement (announcement).
 * Andre Engels joined the Mobile team as data analyst contractor (announcement).
 * Howie Fung, previously Senior Product Manager, was promoted to Director of Product Development, a newly formed group within the Engineering department (announcement).
 * Chris MacMahon joined the Platform engineering team as Quality Assurance Lead Engineer (announcement).

Site infrastructure

 * Data Centers — Work continued on building up the EQIAD datacenter in Virginia. We added a new bastion host, a ganglia server, new dataset servers and upgraded the database servers, with a new chained replication topology and a heartbeat-based replication monitoring. We upgraded mailman and migrated it to a new server in EQIAD from ESAM. We have also successfully tested the new thumbnail system and the text squid implementation, that we'll start rolling out fully in February. As for our Tampa datacenter, we have been upgrading the database servers as well as adding new ones for redundancy and capacity needs. We have at least 4 databases per cluster now (except s5, where the 4th one is being added). At the same time, we have retired 40 old servers, freeing up 2 racks of  space and the much needed IP addresses. The retired servers will be available for donation soon.


 * Media Storage — Performance testing of Swift continued in January. We confirmed that performance degrades above roughly 10 million objects in a bucket, so we adjusted both the Swift middleware and Mediawiki support to allow sharded containers. Our current plan is to shard the Commons and English Wikipedia containers.  The test hardware for object storage arrived; we validated that it works as expected and have ordered what will be the production hardware and expect it to arrive and go into service towards the end of February. We have started populating the current cluster in Tampa with thumbnails in preparation for putting it into production service.


 * HTTPS — HTTPS work is still on hold in favor of other projects. We did have some activity thanks to volunteer Abe Music who fixed a portion of our UDP logging module for Nginx. There is one more fix still needed before HTTPS page views are properly tracked in our statistics. The following outstanding issues have been fixed: nagios (replaced self-signed cert), upload (served wrong cert via IPv6), integration (wrong cert / certificate chain), and resolved in December already: noc and planet. There are remaining issues related to: jobs, status and mail. The wiki table of HTTPS-less domains has been updated, details can be found in the edit history.

Testing environment

 * Wikimedia Labs — To keep up with project growth, the virtualization controller  has been converted into a compute node. Doing so let   and   join the instance storage, doubling the filesystem storage available. The additional compute node also allows Labs to grow by up to another 30 instances. An old ruby gateway server  was converted into the virtualization controller . A number of projects were added or moved to Labs, including incubator, ganglia, deployment-prep, globaleducation, a number of mobile projects, and a bunch more. Labs was very useful during the SF hackathon. A number of projects were created, implemented, and demoed using Labs during the hackathon. We also had a project created, implemented, tested, and deployed to production during the hackathon. We are still waiting for the gluster storage to arrive for volume storage; it should arrive early February. There are now 46 projects, 80 instances and 103 users.

Backups and data archives

 * Data Dumps — A problem with the rsync to our mirror site was located and fixed. Another organization agreed to mirror the dumps as well, and we are waiting for their server to come online. Back issues of dumps from 2002 through 2006 were made available, for folks interested in historical data. New hardware has arrived in our Virginia datacenter, and we'll be copying all dumps over there as soon as it's ready. We're thinking about how to provide image dumps in some fashion, even if we don't keep local copies of the dumps or they are not run on a regular basis. We also cleaned up the dumps documentation and drafted this year's development plans. Finally, we have a contractor, Christian Aistleitner, who will be working on a test suite for dump generation.

Other news

 * Some users complained of slowness and pages not rendered on Wikipedia in Occitan. Domas Mituzas, one of our volunteers, helped fix the issue temporarily. After further investigation, he found that the root cause was badly constructed templates, which were hoarding up server memory RAM. Fixes are planned to put in place to prevent this kind of problem from surfacing again.
 * Some US-based users of Wikipedia experienced slow page rendering time for 10–15 minutes on January 20, 2012; this was due to the bits.wikimedia.org servers in EQIAD being overloaded. The issue was investigated and quickly resolved.

Mobile

 * Android Wikipedia App — The Mobile team released the first version of the Wikipedia Android application into the Android Market. In just over three weeks, we've had over 900,000 downloads, became the #1 search result for "Wikipedia", became the #1 trending app, and received a consistent 4/5 stars in the Android Market. We released two minor updates to fix bugs, and are processing user feedback to guide our next version.


 * WikipediaZero — We launched our first demo version of WikipediaZero for carrier testing. While there is till much work to be done in order to integrate with as many carriers as we'd like to see, we're already starting to make progress on how to simplify our implementations.


 * Wikipedia over SMS/USSD — Patrick Reilly, along with the PraeKelt Foundation, worked on a demo instance of a SMS/USSD gateway to access Wikipedia. We're hoping to have a complete demo in time for Mobile World Congress next month.


 * GPS Storage/Retrieval — An early prototype of the GPS storage retrieval API went live this month. We still have a large to-do list in order to roll it out in production, but it's showing great early stage progress.


 * FeaturedFeeds — Max Semenik, with the help of Arthur Richards, deployed the first version of FeaturedFeeds to production. Wikimedia communities can now make use of these RSS feeds to better surface their content to other applications. A list of existing feeds is available on the Toolserver.

Offline

 * Kiwix UX initiative — We have finished the second round of the UX initiative. Focus is now on porting Kiwix to Android/ARM, and we expect to release a first beta version for the end of February. Another goal is to reduce the show-stopper bug list for final release of Kiwix Desktop v0.9.

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.