Wikimedia Release Engineering Team/Checkin archive/20180611

= 2018-06-11 =

Vacations/Important dates

 * https://office.wikimedia.org/wiki/HR_Corner/Holiday_List
 * How to do it


 * June 19: WMF Monthly Holiday - https://en.wikipedia.org/wiki/Juneteenth
 * June 18-22: Dan – Vacation
 * July 2: Greg's BDay!!!
 * July 4: US Holiday
 * July 16: Mukunda's bday.....funtimes
 * August 15: WMF Monthly Holiday
 * Middle of August...a few days somewhere - thcipriani

Train

 * Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-fmcvjrkfvvzz3gxavs3a&statuses=open%28%29&group=none&order=newest#R


 * May 21 - wmf.5 - Mukunda
 * May 28 - wmf.6 - Tyler
 * June 04 - wmf.7 - Tyler
 * June 11 - wmf.8 - Dan (with Tyler doing Thursday)
 * June 18 - wmf.9 - Dan (no train, SRE summit)
 * June 25 - wmf.10 - Dan
 * July 02 - wmf.11 - Zeljko - no train, Fourth of July
 * July 09 - wmf.12 - Zeljko
 * July 16 - wmf.13 - Antoine
 * July 23 - wmf.14 - Antoine

SoS

 * May 21 - Chad
 * May 28 - Mukunda
 * June 04 - Mukunda
 * June 11 - Tyler <-
 * June 18 - Tyler
 * June 25 - Tyler
 * July 02 - Dan
 * July 09 - Dan
 * July 16 - Zeljko
 * July 23 - Zeljko
 * July 30 - Antoine
 * August 06 - Antoine

Updates

 * Train: Decided: Dan this week


 * Train EU time:
 * EU SWAT 1300-1400 CEST (1100-1200 UTC), 2 hours earlier than now
 * break 1400-1500 CEST (1200-1300 UTC), my usual lunch time
 * train 1500-1700 CEST (1300-1500 UTC)
 * break 1700-1800 CEST (1500-1600 UTC)
 * Puppet SWAT 1800-1900 CEST (1600-1800 UTC)


 * SWAT window on Wednesday
 * bad timing (with SoS)
 * was moved back for the pre-Train sanity hour
 * move it back to 11 Pacific?
 * DECIDED: move back to 11 Pacific time


 * Backfill needs/desires
 * Developer Productivity JD: https://docs.google.com/document/d/1Jnt9gCZ0zk0lC_aYYzzE9x3JxnLsxhm6QKAjXCaGrho/edit
 * proposal: add a list of keywords of things (software/languages) we use
 * Greg will make a Sr Engineer focused version


 * https://phabricator.wikimedia.org/T196840 - fyi mmodell

Offsite notes and action items

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Offsites/2018-05-Barcelona/Notes

Data Data Data

 * Talk with Analytics - JR
 * Talk with CE/Bitergia - JR
 * Explore Bitergia - JR
 * Identify data sources we want to collect - RelEng (who know what systems)
 * https://etherpad.wikimedia.org/p/releng-data-sources
 * TODO: Greg: email the etherpad out to the team list
 * Erik Bernhardson / Guillaume Lederrey
 * After we get our list of data sources talk to them

SWATs/Trains

 * TODO: Tyler reasses scap swat in mw-config from Mukunda
 * Look into parsing scap messages for known patterns and pulling out the data
 * Look into enabling scap start/done
 * This should be trivial
 * Look into recording if mwdebug was used during the deploy (eg: 'scap stage')
 * H/Now will we get time for this?
 * Have Mukunda do a couple weeks of SWATs
 * Mukunda has a lot to say about this subject.... writeup incoming

TODO tl;dr: Tyler taskify the above and talk to Mukunda
 * https://phabricator.wikimedia.org/T193311
 * https://phabricator.wikimedia.org/T196411
 * work in progress docs: https://wikitech.wikimedia.org/wiki/SWAT_deploys/scap_swat

Staging

 * Greg to talk with Deb about what to do next with talking to Victoria
 * Done
 * Greg to figure out how we can better market what we are accomplishing (eg "monthly showcase")
 * TODO
 * Get a k8s cluster from SRE for CI to deploy to.
 * Dan/Tyler discuss later today
 * potentially poke/get a committment on a timeline soon

Scrum of Scrums

 * Greg to copy to etherpad after meeting: https://etherpad.wikimedia.org/p/Scrum-of-Scrums

Release Engineering

 * Blocking
 * Blocked
 * Updates
 * Quarterly cross-dependencies
 * Updates
 * Quarterly cross-dependencies
 * Quarterly cross-dependencies

Release Engineering

 * Blocking
 * Blocked
 * Updates
 * Please take the Beta Cluster survey:
 * https://lists.wikimedia.org/pipermail/wikitech-l/2018-May/090049.html
 * https://docs.google.com/forms/d/e/1FAIpQLScw01fMHA9wNTUtp-_iIOdhmQpm0L38MHK-RS_MTvdnlro0lA/viewform
 * Heads up: There will be more people in the normal MW Train deployment rotation (namely: Antoine, Zeljko, and Dan to start) and we'll be doing some Train deployments during EU hours some weeks. Exact timing TBA (soon).
 * Heads up: There is a new primary deployment host, deploy1001.eqiad.wmnet (tin is gone)
 * Reminder: just use deployment.eqiad.wmnet for ssh
 * Quarterly cross-dependencies
 * Reminder: just use deployment.eqiad.wmnet for ssh
 * Quarterly cross-dependencies

Train status and happenings

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Roles#Train_Conductor

Past week status updates

 * All of it in table form: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201718Q4

Program 1: Outcome 5: Objective 1: Maintain existing shared Continuous Integration infrastructure

 * Migrate away from Nodepool -
 * Migrate MediaWiki PHPUnit tests to Shipyard (docker-based CI) (~40% of Nodepool usage) -
 * Add Composer support to Blubber -
 * Add Python/Tox support to Blubber -
 * Add Ruby/Gem/Bundler/Rake support to Blubber -

Program 3: Outcome 1: Objective 2: Identify and find stewards for high-priority/high use code segment orphans

 * Broad role out of Code Stewardship model.
 * Update Maintainers/Developers page with currently known Code Stewards.
 * Use Code Stewardship review process to address gaps.
 * Deploy dashboard of Code Stewardship Coverage


 * Reviewed feedback and met with Toby and Victoria
 * Zotero issue seems to be addressed

Program 3: Outcome2: Objective 2: Define and implement a process to regularly address technical debt across the Foundation

 * Roll out of technical debt reduction approaches.
 * Identify early adopter engineering teams to test approaches.


 * Reviewed Platform technical debt tasks in phab
 * Reviewed general technical debt tasks in phab

Program 6: Outcome 2: Objective 2: Prove viability of testing staged service containers alongside MediaWiki extension containers

 * Add Composer support to Blubber -
 * Small, standalone, MediaWiki containers built using Blubber
 * Limited scope with: Debian Stretch, php7, composer dependencies, Mariadb, Apache (or last two with standalone containers)
 * Build MW base containers upon branch cut at master branch point from core
 * Build ext specific containers using MW image as base

Few discussions last week:
 * Math extension post-merge job
 * Need to break-up further (see bullet points)
 * CI namespace for staging k8s
 * SRE asked for a task last Thursday
 * Hopefully Soon™

Program 1: Outcome 1: Objective 1: Scap (Tech Debt Sprint FY201718-Q2)

 * workboard

Program 1: Outcome 6: Milestone 2: Maintain Phabricator

 * Streamline logspam workflows by adding some integration with phabricator
 * Store git-lfs (and other phab uploads) in swift:

** https://phabricator.wikimedia.org/T190572
 * Making some progress on a disaster recovery plan for phabricator:

Antoine

 * What I plan to do this week
 * Migrate mediawiki-extensions jobs
 * What I'm blocked on
 * Whatever tests is going to end up failling :/ Eg Selenium triggering a bug in mediawiki https://phabricator.wikimedia.org/T191537
 * Other?

Dan

 * What I plan to do this week
 * Train
 * Docker image build for Math extension
 * What I'm blocked on
 * Other?
 * Other?

Greg

 * What I plan to do this week
 * A couple high prio HR/Fiscal year things
 * Announce EU Train window and deploy schedule changes
 * Make a Senior Eng focused version of the DevProd JD
 * email the "what data sources do you know of" etherpad out to the team list
 * What I'm blocked on
 * nothing yet
 * Other?

Jean-Rene

 * What I plan to do this week
 * More on Search platform and platfom technical tech
 * Q4 stewardship reviews
 * Q3 stewardship sync-ups
 * offsite tasks
 * What I'm blocked on
 * Other?
 * Other?

Mukunda

 * What I plan to do this week
 * Fix phabricator importing new gerrit meta commits
 * https://phabricator.wikimedia.org/D1067
 * Work on scap swat stuff (documentation, clean up cli api)
 * https://phabricator.wikimedia.org/T196411
 * support git-lfs stuff as needed
 * Try to get some movement on phabricator+swift puppetization
 * https://phabricator.wikimedia.org/T196411
 * What I'm blocked on
 * Waiting on filippo to review puppet patches for swift
 * Should be nearly ready to merge and the patches got some review early this morning.
 * https://gerrit.wikimedia.org/r/c/operations/puppet/+/432528
 * Other?

Tyler

 * What I plan to do this week
 * "Train"ing
 * Math extension pipeline sense-making (see bullet points on task)
 * Scap 3.8.2-1 help/fallout/whatever's needed
 * What I'm blocked on
 * None?
 * Other?
 * Filed Automate the Train

Zeljko

 * What I plan to do this week
 * T190710 Minerva Ruby and Node.js browser tests running side by side
 * T194252 Configure the CI job that runs WikibaseLexeme's browser tests against beta wikidata
 * What I'm blocked on
 * Other?
 * T195496 mediawiki-core-qunit-selenium-jessie sometimes fails for extensions - Antoine and Anomie made great progress https://gerrit.wikimedia.org/r/c/mediawiki/core/+/439289
 * T195496 mediawiki-core-qunit-selenium-jessie sometimes fails for extensions - Antoine and Anomie made great progress https://gerrit.wikimedia.org/r/c/mediawiki/core/+/439289

Team Kanban Board Review and Triage

 * closed and touched in the 7 days
 * No update for 4 weeks
 * No update for 3 weeks
 * No update for 2 weeks
 * No update for 1 week
 * All Open
 * Review To Triage column of #releng
 * Assigned
 * Unassigned

Once / month-ish review of backlog(s)

 * releng Review To Triage column of #releng
 * releng-kanban Review unassigned in kanban
 * releng-kanban Review 'backlog' colum of -kanban
 * releng-next - Review for things we need to put on our kanban backlog
 * releng-backlog - oh my, the huge backlog of things...

Kanban stats

 * Burnup chart