Wikimedia Platform Engineering/MediaWiki Core Team/Check-ins/20140113

Who: Brad, Chad, RobLa, Aaron, Chris, Nik, Bryan, Ori, Antoine

Escalations
Need post-mortem for CategoryTree related downtime earlier today. Greg to followup with Sam

Bug 59957: Gerrit host has weird CPU usage Gerrit outage over the weekend. Chad will investigate + update bug.

Search

 * lsearchd hasn’t updated in a week (bug 59979)
 * Chad and Nik have looked, somewhat stumped.
 * Going to research further, will prod Ori/Tim/ops as needed
 * We’re poking Leslie on IRC now
 * Cirrus now secondary search backend on enwiki
 * enwiki job queue backed up (not our fault, but we noticed)
 * 1.7mil htmlCacheUpdate jobs and rising (+30k since I started watching it this morning)
 * Raising job runners by 1 per box: https://gerrit.wikimedia.org/r/#/c/107189/
 * Seems to have helped, numbers going down again (-45k, roughly?)

DevOps

 * Aaron documented deployment changes in Jan 9 e-mail to engineering@
 * Bryan to sit in with Max tomorrow
 * Aaron looking into new magic word handling and job queue issues

Performance

 * Aggregated and plotted page load timing data for last 10 months: .
 * TODO: Automate the analysis and produce a weekly report to engineering@ / wikitech-l.
 * Presented at January Metrics Meeting: [[File:Site performance update - January 2014 Wikimedia Foundation monthly Metrics & Activities Meeting.pdf]]
 * Lots of UniversalLanguageSelector analysis and back-and-forths prompted by bug 59958. Reviewed architecture in bug 56433#c11, culminating in Erik’s decision in 56433#c22 to disable automatic font loading, which is a good outcome.
 * Tracked down core test failure under HHVM in bug 55532#c2. Good news: fixed upstream.
 * Tim’s xhprof patch merged to HHVM: https://github.com/facebook/hhvm/pull/1488
 * Andrew Otto ready to configure profiling stats endpoint on bits, waiting on me to e-mail ops@ to get a nod of approval from Faidon or Mark.
 * Still thinking about performance guidelines for developers.

Beta / CI

 * Extensions on beta no more self deploying bug 49847
 * mediawiki/extensions.git no more updated by Gerrit. Need to write some script to make it happen for us.

Security

 * Release via M&M today - https://bugzilla.wikimedia.org/show_bug.cgi?id=59830

Scholarships

 * ~80 applications received so far.
 * No serious bugs found but several small changes to text and validation have been requested and deployed.

Quarterly review
https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Quarterly_review,_January_2014

Architecture Summit
https://www.mediawiki.org/wiki/Architecture_Summit_2014

https://www.mediawiki.org/wiki/Architecture_Summit_2014/Straw_poll Things not yet on the agenda that should be:
 * Ori: Code review standards
 * RobLa; RFC lightning round