Wikimedia Release Engineering Team/Checkin archive/20181119

= 2018-11-19 =

Vacations/Important dates

 * https://office.wikimedia.org/wiki/HR_Corner/Holiday_List
 * How to do it


 * November 19 - Jeena's start date
 * November 21 - Dan vacation (road trip!)
 * November 22+23 - Holidays (Thanksgiving)
 * November 25-december 2nd: Mukunda vacation (in California ahead of the offsite)
 * Week of December 3rd - Team offsite
 * December 24-28 - Holidays (Christmas)

Train

 * Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-fmcvjrkfvvzz3gxavs3a&statuses=open%28%29&group=none&order=newest#R


 * Oct 08 - wmf.25 - Dan (No train due to DC switchover)
 * Oct 15 - wmf.26 - Mukunda (last 1.32 wmf.XX release, 1.33 starts the next week)
 * Oct 22 - wmf.1 - Mukunda (warning, TechConf happening, ping Greg if you need responses from anyone there...)
 * Oct 29 - wmf.2 - Tyler
 * Nov 05 - wmf.3 - Tyler
 * Nov 12 - wmf.4 - Antoine
 * Nov 19 - wmf.5 - No Train (Thanksgiving) <
 * Nov 26 - wmf.6 - Antoine
 * Dec 03 - wmf.7 - No Train (Offsite)
 * Dec 10 - wmf.8 - Zeljko
 * Dec 17 - wmf.9 - Zeljko
 * Dec 24 - wmf.10 - No Train (Holiday break)
 * Dec 31 - wmf.11 - No Train (Holiday break)
 * Jan 07 - wmf.12 - Dan
 * Jan 14 - wmf.13 - Dan
 * Jan 21 - wmf.14 - Mukunda
 * Jan 28 - wmf.15 - No Train (All Hands)
 * Feb 04 - wmf.16 - Mukunda
 * Feb 11 - wmf.17 - Tyler
 * Feb 18 - wmf.18 - Tyler
 * Feb 25 - wmf.19 - Antoine

SoS

 * Zeljko for ever :)

Welcome Jeena!

 * Hi Jeena :)
 * Welcome!

December Offsite
Details:
 * Week of December 3rd
 * At the Queen Mary hotel in Long Beach
 * Deb T will be facilitating
 * REMINDER: Deadline to book travel is Nov 8th!
 * Topics!
 * https://etherpad.wikimedia.org/p/RelEng-Offsite-201811-Topics

Activity decision: Both the Aquarium of the Pacific and the USS Iowa on Friday.
 * https://en.wikipedia.org/wiki/USS_Iowa_(BB-61)
 * https://en.wikipedia.org/wiki/Aquarium_of_the_Pacific

All Hands

 * Registration: https://office.wikimedia.org/wiki/All_hands/2019/Registration
 * Needed for everyone
 * NOTE: There's a way to request a hotel room for semi-local people (commutes longer than 1.5 hours)

OMG Surveys

 * Engagement survey
 * Please take it! :) You should have an email from "Culture Amp".


 * Internal Support for Open Source Software
 * Survey: https://wikimedia.qualtrics.com/jfe/form/SV_1Ojwuii3rgtawuN
 * Announcement: https://groups.google.com/a/wikimedia.org/forum/#!topic/foundation-optional/N5y0xCHnmYY

Incoming Triage/Needs attention

 * deploy1001:/srv/mediawiki out of date?
 * https://phabricator.wikimedia.org/T207602
 * Found because the Security team noticed that a previously deployed security patch was no longer deployed, should sync up with them this week about that (Reedy or Brian)
 * See: https://phabricator.wikimedia.org/T207600
 * 2018-10-22: no idea, thcipriani will look, I guess
 * 2018-10-29: scap updated, needs release this week
 * 2018-11-05:
 * Need to poke Reedy re:T207600
 * scap still needs release - mukunda will take care of it
 * 2018-11-19:
 * Poked Reedy re:T207600
 * Scap 3.8.9-1 running in beta


 * deployment-prep region migration
 * See email with same subject on releng@lists
 * Question: incrementally or not?
 * looks like "however Andrew wants to do it"
 * REMINDER: send an email update to wikitech-l@/qa@ with the planned timeline/outage
 * 2018-10-29: ACTION: Tyler to reply saying "take it away, andrew, and when are you going to do it?"
 * 2018-11-05: Email response ✅ -- blocking task from Krenair https://phabricator.wikimedia.org/T208101 -- Dan and Mukunda graciously volunteered ;)
 * 2018-11-19: squeeze in the migration this week? Yes. Mukunda to work on that with Andrew and use paring with Tyler to discuss any breakages.


 * Migrate the Integration cloud project to eqiad1-r
 * https://phabricator.wikimedia.org/T208803
 * 2018-11-12: Need a point person to work with Andrew on this
 * 2018-11-19: Tyler and Andrew migrated a few, no issues so far. integration-publishing migrated. castor02 can be migrated the same ways as other slaves, culprit might happen see task for details.


 * Migrate the contintcloud cloud project to eqiad1-r
 * Nodepool stopped on Thursday. Antoine asked for project to be deleted.

Scrum of Scrums

 * Greg to copy to etherpad after meeting: https://etherpad.wikimedia.org/p/Scrum-of-Scrums

Incoming from last week

 * Blocking: nothing...

Release Engineering

 * Blocked by:
 * Blocking:
 * Updates:
 * Train Health:
 * Last week: 1.33.0-wmf.4 deployment blockers https://phabricator.wikimedia.org/T206658
 * This week: No train due to the Thanksgiving Holiday
 * Next week:
 * Log Health:
 * Code Health:
 * Bring in 'da noise, bring in defunct. It's a zombie party! Peering into the CI cauldron – October 2018 https://phabricator.wikimedia.org/phame/post/view/127
 * Code Health:
 * Bring in 'da noise, bring in defunct. It's a zombie party! Peering into the CI cauldron – October 2018 https://phabricator.wikimedia.org/phame/post/view/127
 * Bring in 'da noise, bring in defunct. It's a zombie party! Peering into the CI cauldron – October 2018 https://phabricator.wikimedia.org/phame/post/view/127

Callouts

 * Release Engineering
 * Blubber build with Blubber
 * Tyler: Gerrit is alright. Got JavaMelody for monitoring now. InitialiseSettings.php is too big.
 * Zeljko/Gehel: https://sonarcloud.io/organizations/wmftest/projects
 * Antoine: Nodepool stopped yeah! #technical-debt

Train status and happenings

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Roles#Train_Conductor

TEC1 (Maint): Outcome 1 / Output 1.1

 * GOAL: Release MediaWiki 1.32
 * WHO: Mukunda, (Tyler on backup)


 * 2018-11-05: Mukunda to release MediaWiki 1.32 RC.0
 * rc.0 Done

TEC1 (Maint): Outcome 1 / Output 1.1

 * GOAL: Determine the procedure and requirements for an automated MediaWiki branch cut.
 * WHO: Mukunda, Tyler, Antoine


 * mukunda + tyler to pair on deploy notes this week

TEC3 (Pipeline): Outcome 1 / Output 1.2

 * GOAL: Formalize the collection of CI infrastructure and tooling metrics
 * WHO: Dan, Antoine

TEC3 (Pipeline): Outcome 2 / Output 2.3

 * GOAL: Develop set of metrics to assess incident reports/post mortems -
 * WHO: Greg, Zeljko

TEC3 (Pipeline): Outcome 3 / Output 3.1

 * GOALS:
 * Adopt more services into Deployment pipeline -
 * Migrate graphoid to the Deployment pipeline
 * Deploy zotero v2 to the Deployment pipeline
 * Deploy blubberoid
 * WHO: Dan, Tyler, Lars


 * Lars created blubber production variant blubber file!
 * Saw movement on zotero this weekend, to prod this week
 * graphoid -- thcipriani to poke this week

TEC12 (DevProd): Outcome 2 / Output 2.1

 * GOAL: The Annual Developer Productivity Survey results are synthesized and shared, creating a first year baseline.
 * WHO: Mukunda, Greg


 * 2018-11-13:
 * 45 responses so far
 * 35 from staff
 * 9 volunteer
 * 1 other / 3rd party
 * Overall Satisfaction:
 * Local Development: 3.37 / 5 (67%)
 * Beta Cluster: 3.51 / 5 (70%)
 * Testing: 3.37 / 5 (67%)
 * Code Review: 3.37 / 5 (67%)
 * Deployments 4.0 / 5 (80%)
 * Production Systems: 4.0 / 5 (80%)
 * Tools: 4.0 / 5 (80%)
 * Documentation: 3.1 / 5 (62%)
 * Lots of good feedback given
 * I'm very happy with the results of this survey.

TEC13 (Code Health): Outcome 1 / Output 1.1

 * GOAL: Update/refresh review queue (review process for initial code deployment)
 * WHO: JR

No activitiy.

TEC13 (Code Health): Outcome 2 / Output 2.2

 * GOAL: 5 of the 15 prioritized repositories have at least 1 end-to-end test -
 * WHO: Zeljko

Top 15 defined. Deeper dive into data to see if it supports other course of action.

TEC13 (Code Health): Outcome 2 / Output 2.3

 * GOAL: Assess Platform unit test practices and define improvement plan
 * WHO: JR, Core Platform Team

Planned working session with Corey/Cindy Review Unit testing activities of Core

TEC13 (Code Health): Outcome 3 / Output 3.2

 * GOAL: Core Platform and Search Platform teams are using TDM PoC
 * WHO: JR, Core Platform Team

Planned working session with Corey/Cindy

TEC13 (Code Health): Outcome 3 / Output 3.4

 * GOALs:
 * Identify key Tech Debt areas
 * Put in place Tech Debt management process for PEP
 * WHO: JR, Core Platform Team

Planned working session with Corey/Cindy

TEC13 (Code Health): Outcome 4 / Output 4.1

 * GOAL: Metrics defined and deployed for all 4 Code Health areas.
 * WHO: JR, Code Health Metrics Working Group

Post-merge SonarQube scanner setup and running.

TEC3 (Pipeline): Outcume 2 / Output 2.3

 * GOAL: Outline options for managing incident reports creation, follow-ups, and analysis
 * WHO: Greg, Mukunda, Zeljko
 * Task: https://phabricator.wikimedia.org/T208632


 * 2018-11-05: mukunda to make some comments

Gerrit

 * Problem shortly after 2.15.6 upgrade
 * Antoine noticed a bunch of bots in logs
 * Mukunda + tyler did troubleshooting
 * Installed javamelody monitoring: https://gerrit.wikimedia.org/r/monitoring
 * Added a robots.txt that blocks indexing gitiles
 * seems stable
 * Followup action adding prometheus exporter to gerrit https://phabricator.wikimedia.org/T184086
 * 2.15.7 out
 * let's do the prep work at the offsite!

Antoine

 * What I plan to do this week
 * Maintenance
 * Doc about MediaWiki extensions testing. Hoping to publish draft by end of week
 * (stretch) fill tasks to migrate jobs from the old legacy Jenkins slaves
 * (stretch) push to get Nodepool decomissioned entirely
 * What I'm blocked on
 * Magic?!
 * Other?

Dan

 * What I plan to do this week
 * Prometheus! (for realz)
 * Be Jeena's onboarding buddy
 * What I'm blocked on
 * Knowing stuff
 * Being gone on Wednesday
 * Other?

Greg

 * What I plan to do this week
 * Onboarding :)
 * some workboard grooming (is a hope/desire)
 * Staging discussion tomorrow morning at 7am
 * warning on future meeting with Victoria
 * Review Zeljko's blog post
 * What I'm blocked on
 * determining when I should head back south/home :)
 * Other?

Jean-Rene

 * What I plan to do this week
 * Code Health Newsletter
 * RoO/Review queue


 * What I'm blocked on
 * Not blocked, but needed to postpone work with Core Platform Team to next week.
 * Other?

Jeena

 * What I plan to do this week
 * https://phabricator.wikimedia.org/T209722
 * What I'm blocked on
 * Everything :)
 * Other?
 * Computer arrived!

Lars

 * What I plan to do this week
 * Blubberfile for Blubber
 * Try to understand how Blubberoid could be deployed via delivery pipeline
 * What I'm blocked on
 * nada
 * Other?
 * nada

Mukunda

 * What I plan to do this week
 * Help andrewbogot with moving deployment-prep to the new cloud region
 * Assuming the above goes smoothly, release mediawiki 1.32.0-rc.1
 * What I'm blocked on
 * Other?
 * Other?

Tyler

 * What I plan to do this week
 * pairing with Jeena if she has time this week
 * GPG signing and pwstore setup
 * Pipeline image versioning update
 * Graphoid poking
 * Move to no-helm-test pipeline to create base image post merge
 * Pairing on deploy notes
 * Gerrit 2.16 stable branch
 * Patches for docker on releases-jenkins
 * What I'm blocked on
 * Other?
 * Other?

Zeljko

 * What I plan to do this week
 * T199133 Find top 15 target projects that could use Selenium tests to prevent incidents
 * You'll know more than you want to know by the end of the meeting
 * T207046 Code health metrics spike
 * I might need somebody's help (Antoine?) on Gerrit bots
 * What I'm blocked on
 * Other?
 * Other?

Team Kanban Board Review and Triage

 * closed and touched in the 7 days
 * No update for 4 weeks
 * No update for 3 weeks
 * No update for 2 weeks
 * No update for 1 week
 * All Open
 * Review To Triage column of #releng
 * Assigned
 * Unassigned

Once / month-ish review of backlog(s)

 * releng Review To Triage column of #releng
 * releng-kanban Review unassigned in kanban
 * releng-kanban Review 'backlog' colum of -kanban
 * releng-next - Review for things we need to put on our kanban backlog
 * releng-backlog - oh my, the huge backlog of things...

Kanban stats

 * Burnup chart