Wikimedia Engineering/Report/2012/January

Major news in January include:
 * SOPA

Hover your mouse over the green question marks to see the description of a particular project.

Recent events

 * SOPA


 * San Francisco hackathon (21–22 January 2012, San Francisco, California, USA) —

Upcoming events

 * Pune hackathon (10–12 February 2012, Pune, India) —


 * GLAMcamp DC (10–12 February 2012, Washington, D.C., USA) —

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.


 * Developers and engineers:
 * Senior Software Engineer Front-end
 * Interaction Designer
 * Systems Engineer (Data Analytics)
 * Software Developer (Back-end, Data Analytics)
 * Software Developer (Rich Text Editing, Features)
 * Software Developer (Front-end)
 * QA Lead
 * Software Developer (Mobile)
 * Software Security Engineer
 * Operations Engineer (Labs)


 * Management & Product:
 * Director of Features Engineering


 * Requests for proposals:
 * Executive Dashboard - Analytics — Help us improve and centralize the dashboard summarizing the most important data about the Wikimedia movement to understand overall community health.
 * Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

 * Andrew Otto joined the Platform engineering team as Software Developer for Analytics ([//lists.wikimedia.org/pipermail/wikitech-l/2012-January/057430.html announcement]).
 * Fabrice Florin joined the Strategic Product team as Product Manager for New Editor Engagement (announcement).
 * Andre Engels joined the Mobile team as data analyst contractor (announcement).

Site infrastructure

 * Data Centers —


 * Media Storage —


 * HTTPS — HTTPS work is still on hold in favor of other projects. We did have some activity thanks to Abe Music fixing a portion of our UDP logging module for Nginx. There is one more fix still needed before HTTPS page views are properly tracked in our statistics.

Testing environment

 * Wikimedia Labs — To keep up with project growth, the virtualization controller (virt1) has been converted into a compute node. Doing so let virt1 and virt4 join the instance storage, doubling the filesystem storage available. The additional compute node also allows Labs to grow by up to another 30 instances. An old ruby gateway server (mobile2) was converted into the virtualization controller (virt0). A number of projects were added or moved to Labs, including incubator, ganglia, deployment-prep, globaleducation, a number of mobile projects, and a bunch more. Labs was very useful during the SF hackathon. A number of projects were created, implemented, and demoed using Labs during the hackathon. We also had a project created, implemented, tested, and deployed to production during the hackathon. We can thank Abe Music for his help with our custom Nginx UDP logging module. He fixed one of two outstanding issues that was skewing our statistics. His fix is now running live on the sites. We are still waiting for the gluster storage to arrive for volume storage. It should arrive early February. There are now 46 projects, 80 instances and 103 users.

Backups and data archives

 * Data Dumps — A problem with the rsync to our mirror site was located and fixed.  Another organization agreed to mirror the dumps as well and we are waiting for their server to come on line.  Back issues of dumps from 2002 through 2006 were made available, for folks interested in historical data. New hardware haa arrived in our data center in D.C. and we'll be copying all dumps over to a server there as soon as it's ready. We're thinking about how to provide image dumps in some fashion, even if we don't keep local copies of the dumps or they are not run n a regular basis.  We also cleaned up the dumps documentation on wikitech and drafted this year's development plans. And finally, we have a contractor, Christian Aistleitner, who will be working on a test suite for dump generation. Yay!

Other news

 * We had some intermittent performance problem with blogs.wikimedia.orf when there was a surged in traffic during the days leading to SOPA blackout. The team upgraded the hardware, added caching (Varnish and Memcached) as well as tuned the software for pagination. Those actions helped.
 * Some users were complaining of slowness and pages not rendered to oc.wikipedia.org. Domas, one of our volunteers, who was on line helped fixed that problem temporarily. Subsequently on further investigation, he found the root cause was because of badly constructed templates which were hoarding up server memory RAM. Some fixes are now in place to prevent such templates fom hoarding up RAM spaces.
 * Some US users of WP site experienced slow page rendering time for about 10 - 15 minutes (from 23:18 UTC to 23.33 UTC) on 20th Jan, 2012, because the Ashburn (EQIAD) bits.wikimedia.org servers were overloaded. Most however had incorrectly formatted wiki pages due to missing javascript and css. Site issue January 2012

Mobile

 * Mobile Research —


 * Android Wikipedia App —


 * WikipediaZero —


 * GPS Storage/Retrieval —


 * Featured Article RSS —

Offline

 * Kiwix UX initiative —

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.