Wikimedia Engineering/Report/2012/January

Major news in January include:
 * Tech support for the black-out protesting SOPA & PIPA;
 * The release of the official Wikipedia Android app;
 * A new beta cluster for Wikimedians to test upcoming software before it's deployed to production;

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * SOPA
 * CongressLookup extension


 * San Francisco hackathon (21–22 January 2012, San Francisco, California, USA) — More than 90 participants learned and hacked during this outreach-focused developers week-end. Sumana Harihareswara and several other WMF staffers worked with the coworking venue pariSoma to organize the event.  The teams of participants demonstrated more than a dozen projects.  The demos, a speech by Erik Moller, and tutorials in mobile, Gadgets, and the web API were recorded, and recordings are forthcoming.

Upcoming events

 * Pune hackathon (10–12 February 2012, Pune, India) —


 * GLAMcamp DC (10–12 February 2012, Washington, D.C., USA) —

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Senior Software Engineer Front-end
 * Interaction Designer
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer
 * Operations Engineer (Labs)


 * Management & Product:
 * Director of Features Engineering


 * Requests for proposals:
 * Executive Dashboard - Analytics — Help us improve and centralize the dashboard summarizing the most important data about the Wikimedia movement to understand overall community health.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Andrew Otto joined the Platform engineering team as Software Developer for Analytics ([//lists.wikimedia.org/pipermail/wikitech-l/2012-January/057430.html announcement]).
 * Fabrice Florin joined the Strategic Product team as Product Manager for New Editor Engagement (announcement).
 * Andre Engels joined the Mobile team as data analyst contractor (announcement).
 * Howie Fung, previously Senior Product Manager, was promoted to Director of Product Development, a newly formed group within the Engineering department (announcement).

Site infrastructure

 * Data Centers — Work continued on building up the EQIAD datacenter in Virginia. We added a new bastion host, a ganglia server, new dataset servers and upgraded the database servers, with a new chained replication topology and a heartbeat-based replication monitoring. We upgraded mailman and move the server from ESAM to EQIAD.  We have also successfully tested the new thumbnail system and the text squid implementation, that we'll start rolling out fully in February. As for our Tampa datacenter, we have been upgrading the database servers as well as adding new ones for redundancy and capacity needs. We have at least  4 databases per cluster now (except s5, which the 4th one is being added). At the same time, we are have retired 40 old servers, freeing up 2 racks of  space and the much needed IP addresses. The retired servers will be available for donation soon.


 * Media Storage — Performance testing of Swift continued in January.  We confirmed that performance degrades above roughly 10 million objects in a bucket, so adjusted both the Swift middleware and Mediawiki support to allow sharded containers.  Our current plan is to shard the commons and english wiki containers.  The test hardware for object storage arrived - we validated that it works as expected and have ordered what will be the production hardware and expect it to arrive and go into service towards the end of February.  We have started populating the current cluster in Tampa with thumbnails in preparation for putting it into production service.


 * HTTPS — HTTPS work is still on hold in favor of other projects. We did have some activity thanks to Abe Music who fixed a portion of our UDP logging module for Nginx. There is one more fix still needed before HTTPS page views are properly tracked in our statistics.

Testing environment

 * Wikimedia Labs — To keep up with project growth, the virtualization controller  has been converted into a compute node. Doing so let   and   join the instance storage, doubling the filesystem storage available. The additional compute node also allows Labs to grow by up to another 30 instances. An old ruby gateway server  was converted into the virtualization controller . A number of projects were added or moved to Labs, including incubator, ganglia, deployment-prep, globaleducation, a number of mobile projects, and a bunch more. Labs was very useful during the SF hackathon. A number of projects were created, implemented, and demoed using Labs during the hackathon. We also had a project created, implemented, tested, and deployed to production during the hackathon. We are still waiting for the gluster storage to arrive for volume storage; it should arrive early February. There are now 46 projects, 80 instances and 103 users.

Backups and data archives

 * Data Dumps — A problem with the rsync to our mirror site was located and fixed. Another organization agreed to mirror the dumps as well, and we are waiting for their server to come online. Back issues of dumps from 2002 through 2006 were made available, for folks interested in historical data. New hardware has arrived in our Virginia datacenter, and we'll be copying all dumps over there as soon as it's ready. We're thinking about how to provide image dumps in some fashion, even if we don't keep local copies of the dumps or they are not run on a regular basis. We also cleaned up the dumps documentation and drafted this year's development plans. Finally, we have a contractor, Christian Aistleitner, who will be working on a test suite for dump generation.

Other news

 * Some users complained of slowness and pages not rendered on Wikipedia in Occitan. Domas Mituzas, one of our volunteers, helped fix the issue temporarily. After further investigation, he found that the root cause was badly constructed templates, which were hoarding up server memory RAM. Fixes are now in place to prevent this kind of problem.
 * Some US-based users of Wikipedia experienced slow page rendering time for 10–15 minutes on January 20, 2012; this was due to the bits.wikimedia.org servers in EQIAD being overloaded. The issue was investigated and quickly resolved.

Mobile

 * Mobile Research —


 * Android Wikipedia App —
 * https://blog.wikimedia.org/2012/01/26/announcing-the-official-wikipedia-android-app/


 * WikipediaZero —


 * GPS Storage/Retrieval —


 * Featured Article RSS —
 * https://blog.wikimedia.org/2012/01/26/featuredfeeds-brings-syndication-feeds-of-featured-wikimedia-content/

Offline

 * Kiwix UX initiative —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.