Wikimedia Platform Engineering/MediaWiki Core Team/Check-ins/20140127

Who: Bryan, Nik, Chad, RobLa, Ori, Greg, Chris, Sam, Aaron, Dan, Brad

Quarterly review
Review action items from quarterly review:
 * Sumana & Ken: follow up on possibility of "has signed an NDA" LDAP group
 * Bryan & Chris: look into 2FA or similar for Logstash Authentication for users
 * Apache + 2FA: https://code.google.com/p/google-authenticator-apache-module/
 * Apache + OpenID: http://auth-openid.sourceforge.net/
 * Apache + LDAP: https://httpd.apache.org/docs/2.0/mod/mod_auth_ldap.html (if none of 2FA options work; ie what we are using now)
 * Chad & Nik: Get Brandon a link to a JSON API
 * [Chad] I forgot about this. JSON for Elasticsearch? Can easily ping re: this.
 * [Nik] He wants the output from MediaWiki but Elasticsearch influences MediaWiki so that should be ok.
 * More benchmarks for HHVM & MediaWiki - characterise & pinpoint & quantify benefits of HHVM so we have a real value proposition for rest of org
 * Mark B to look into this: To collect frontend performance data, would be great to have a varnish kafka topic running on bits varnishes actiing as aggregation point, asks Ori. Not urgent
 * Separate eventlogger load-balancing IP? suggests Faidon
 * Look into provisioning baremetal performance testing infra?
 * maybe just an additional job runner for testing HHVM
 * Faidon & Gabriel: Look into provisioning hardware for the large users of Labs, eg Parsoid
 * Describe what to do in the event of a users/admin settings leak
 * script it?
 * Chad: figure out why we still have AdminSettings lingering around. I killed that years ago.
 * Chris S & Sumana: talk about upcoming training, brainstorm approaches

Search

 * This week we’ll start huwiki
 * Next week we’ll go BetaFeature on enwiki (WOOOOOO!)
 * Nik is working on lots of accuracy and index size and performance issues
 * lsearchd is updating again

Deploy Tooling
We had the crazy meeting on Wednesday. Greg will be posting the pictures and sending out a couple emails to start discussions on a couple of the red post-its.

Performance

 * Impact of switching ULS to opt-in: bytes out on bits varnishes at 1-year low, median page load times down by 200ms - 400ms, 75th percentile page load times down by almost a second.
 * RUM graphs moved to Graphite; added stub dashboard at http://gdash.wikimedia.org/dashboards/frontend/
 * Configured http://performance.wikimedia.org/, will have additional dashboards.
 * WikimediaShopLink DOM reflow fixed: https://bugzilla.wikimedia.org/show_bug.cgi?id=57939

Beta / CI

 * Antoine out :(

Gerrit

 * Upgrading to 2.8.1 this evening (starting 5:30ish SF time)

Security

 * Release tomorrow for bug 60339 (core/pdfhandler). 1.22 bugfixes will be included.

Bug escalation
https://bugzilla.wikimedia.org/show_bug.cgi?id=54193 600-700 times per hour on prod
 * Tim thinks it could be fixed by initializing User sooner https://bugzilla.wikimedia.org/show_bug.cgi?id=41201

Architecture Summit
Followups