Wikimedia Release Engineering Team/Checkin archive/20151123

= 2015-11-23 =

Vacations/Confs/etc
How to do it: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off
 * Nov 24: Mukunda - day off (requested in ADP)
 * Nov 25: Greg - Half day (working that morning)
 * Nov 25: Tyler - Half day (need to update ADP still)
 * Nov 25: Mukunda - half day
 * Nov 26: US HOLIDAY - Thanksgiving
 * Nov 27: US HOLIDAY - Day after Thanksgiving
 * Dec 2?-Jan 0x - Antoine, no clue yet
 * Dec 4th: Greg - disconnected, leaving Thursday evening, returning Sunday :)
 * Dec 2x--XX: Antoine, unscheduled yet
 * Dec 21-Jan 1: Greg - vacation, at least (maybe part of the week before)
 * Dec 22-29: Chad - Christmas
 * Dec 23–25: Tyler — Hopeful, probable, Christmas in Kansas!
 * Dec 24-Jan 3: Dan - Holidays
 * Dec 25: US HOLIDAY - Christmas Day - December 25
 * Jan 1: US HOLIDAY - New Year's Day
 * Jan 4 - 8: WikiDev16 + All Hands
 * Jan 16-18: Chad - another music festival
 * Jan 18: US HOLIDAY - Martin Luther King Day
 * Feb 15: US HOLIDAY - President's Day
 * May 17-(?): Dan - paternity leave :D
 * PO Box for pastries? - Antoine
 * May 30: US HOLIDAY - Memorial Day
 * June-ish: Chad - EDC
 * August: France holiday - because french. :)

Actions from last meeting

 * DONE - Greg: Reword the differential dogfooding goal
 * DONE - Greg: Get perferred physcian form from HR for US staff
 * TODO - Antoine + Mukunda should sit down and talk CI/Harbormaster/Nodepool
 * Mukunda just needs to find time to finally test out harbormaster triggering jenkins jobs.
 * IN PROGRESS - Chad: re 1.26 release: "Going to push on all the assigned ones and find assignees for the remainder this week."
 * Dan to continue to bring up in SoS
 * Release is next week, the 25th of Nov. before thanksgiving (US Nov 26th).
 * TODO - Antoine: re KPIs, do mw core gate-processing time for this quarter, reassess for next quarter if needed
 * TODO - No One Yet: investigate carbon aggregation of stats >1 month old behavior
 * ACTION: Antoine to create a task

WIkiDev16

 * Code Review RFC - https://phabricator.wikimedia.org/T114320
 * Scap3 - https://phabricator.wikimedia.org/T114045

Other

 * Dan: Ask team about ideas for onboarding tasks.
 * https://phabricator.wikimedia.org/tag/engineering_onboarding_template/
 * https://phabricator.wikimedia.org/tag/testing_initiative_2015/ <- add ideas here!

Q3 Goals

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201516Q3

Goals timeline:
 * December 3: Group goal scoped and drafted on mediawiki.org for Technology team.
 * December 10: Group goal + all individual team *drafts completed* on mediawiki.org; discuss at Infra+Tech group and identify dependencies.
 * December 17: individual team goals + group goal *finalized* on mediawiki.org; discuss at Monthly Eng Staff.


 * Some discussion in scap meeting Friday


 * scap3 for mediawiki?


 * Beta Cluster Cleanup  (questionable)
 * Incorporate the work from the previous Staging effort into the current Beta Cluster
 * specific outputs?


 * Differential
 * Garner more early adopters (people and projects)
 * how many?
 * one proj per team (at WMF) in differential
 * reading
 * editing
 * discovery
 * "technology"?
 * operations repos that don't have much CI (eg: deb packages)
 * research? performance?


 * Reduce CI wait time
 * we still need a graph for the wait time that's clear :/
 * https://grafana-admin.wikimedia.org/dashboard/db/releng-zuul

KPIs

 * https://grafana.wikimedia.org/dashboard/db/releng-kpis


 * Log spam:
 * Can we explain the two bumps (up and down) of the INFO lines?
 * RunJobs logging changes? See engineering-l/ops-l? (greg was lazy this morning, will look later)
 * Looking good, actually. The log(n) hides it a bit
 * on 9/12 we were at 1k warning, 2k errors
 * last week we've been between 100-800ish for both
 * we should make some graph that shows that more clearly
 * ACTION: Chad take a look at making the >=Warning graph (mostly done, trying to figure out right log)

New vs Maint time spent

 * https://docs.google.com/a/wikimedia.org/spreadsheets/d/1FI90AefwdLHGzVVdrLS6AxcTcJtLFyX0aQTQfyc88s4/edit

#together

 * Team workboard: to triage: https://phabricator.wikimedia.org/project/board/20/query/TRiVy4zOMdR./
 * Team workboard: only-in-#releng(ish): https://phabricator.wikimedia.org/project/sprint/board/20/query/g2T5.QSLJVRQ/

Scrum of Scrums

 * https://phabricator.wikimedia.org/project/board/64/
 * Blocked on us: https://phabricator.wikimedia.org/maniphest/query/h7YTCBTJsepS/#R

CI Scaling

 * https://phabricator.wikimedia.org/project/board/1010/
 * Quarterly Goal: "CI cluster responds to spike in queued builds by starting and registering additional jenkins slaves" - https://phabricator.wikimedia.org/T111106


 * zuul-cloner being added
 * instances now have git mirror of mediawiki/core and operations/puppet. Will add more depending on size / jobs frequency.
 * Have jobs to clone source code and work from /srv/.
 * Then get a convention of capturing artifacts in /log/** /build/**.

Beta Cluster

 * https://phabricator.wikimedia.org/project/board/497/?order=priority

=> Antoine to summarize discussion with Jaime and fill tasks accordingly
 * Antoine / Jaime chatted about databases performances. Random items:
 * https://phabricator.wikimedia.org/T116793 Investigate slow query logging/digest for Beta Cluster
 * https://phabricator.wikimedia.org/T108255 Enable MariaDB/MySQL's Strict Mode (Antione and Jaime)
 * Migrate to Jessie / MariaDB > 5.5 (currently precise) (with Jaime)
 * Enable performance_schema (just antoine?)
 * Hook db1/db2 in tendril
 * Budget for real hardware, ie have beta cluster DB on labdbXXXX hosts
 * Add them to https://tendril.wikimedia.org


 * Team: might have to watch slow query logs / strict errors etc

Deployment Cabal

 * Main: https://phabricator.wikimedia.org/project/board/349/
 * Scap3: https://phabricator.wikimedia.org/project/board/1449/
 * Quarterly Goal: "Migrate all Service team owned services and MW deploys to scap3" - https://phabricator.wikimedia.org/T109926


 * scap 3.0?
 * no idea / pending production use

Diff[usion|erential] migration

 * https://etherpad.wikimedia.org/p/diffuerential-weekly
 * Gitblit-Deprecate: https://phabricator.wikimedia.org/project/board/46/
 * Quarterly Goal: https://phabricator.wikimedia.org/T111465
 * Gerrit-Migration: https://phabricator.wikimedia.org/project/board/9/


 * Find some Differential Ambassadors for each vertical
 * bd808 for Reading
 * Antoine has arcanist installed! ;)
 * Antoine has arcanist installed! ;)


 * gitblit redirect stuffs
 * chad wants more out of mukunda :)
 * merge the first part (today), still work on the extra bits

Release MediaWiki 1.26

 * https://phabricator.wikimedia.org/project/board/1177/
 * Release date 11/25


 * Do we have all bundled extensions/skins migrated to extension.json/skin.json?
 * MediaWiki 1.26 bundled repo should be state of the art https://phabricator.wikimedia.org/T115392
 * confirmedit is ready for backport

Other Work

 * Mukunda is working on subtree merging mediawiki extensions for long-lived release branch.