Wikimedia Release Engineering Team/Checkin archive/20191002

= 2019-10-02 =

Vacations/Important dates

 * https://office.wikimedia.org/wiki/HR_Corner/Holiday_List
 * How to do it


 * September 23–October 4: JR Vacation


 * October 8: Željko holiday (Independence Day)
 * October 11: Željko conference (https://2019.webcampzg.org/ )
 * October 14 (Indigenous Peoples' Day): US Staff


 * November 11  (Veterans' Day): US Staff
 * November 12–15: TechConf19
 * November 16–20: Offsite
 * November 28–29 (Thanksgiving): US Staff


 * December 6: Lars (Finnish Independence Day)
 * December 25–31 (Christmas): US Staff
 * December 25–26: Lars, Christmas


 * 1 January (New Year's Day): US Staff, Lars
 * 20 January (Martin Luther King Jr. Day): US Staff

Train

 * Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R


 * June 24 - wmf.11 - Jeena (with Mukunda)
 * July 1 - wmf.12 - No train (Fourth of July)
 * July 8 - wmf.13 - Jeena
 * July 15 - wmf.14 - Lars (with Antoine)
 * July 22 - wmf.15 - Lars
 * July 29 - wmf.16 - Brennen (with Tyler)
 * Aug 5 - wmf.17 - Brennen
 * Aug 12 - wmf.18 - No Train (Wikimania)
 * Aug 19 - wmf.19 - Zeljko 😱
 * Aug 26 - wmf.20 - Zeljko 😭
 * Sep 2 - wmf.21 - Antoine
 * Sep 9 - wmf.22 - Antoine
 * Sep 16 - wmf.23 - Mukunda
 * Sep 23 - wmf.24 - Mukunda


 * Sep 30 - wmf.25 - Tyler Dan
 * 7 Oct - wmf.26 - Tyler Dan
 * 17 Oct - 1.35.0-wmf.1 - TBD
 * 21 Oct - wmf.2 - TBD
 * 28 Oct - wmf.3 - TBD
 * 4 Nov  - wmf.4 - TBD
 * 11 Nov  - wmf.5 - TBD (? No train – TechConf ?)
 * Who is available from EngProd + CPT + SRE?
 * 18 Nov  - No train – EngProd offsite
 * 25 Nov  - wmf.7 - Antoine? (? No train – Thanksgiving ?)
 * EU Train week?
 * Or just run with a delayed "Thursday" to the next Monday?

SoS

 * Aug 14 onward: Zeljko
 * SoS on 2019-10-09 is at the same time as Tech All Hands

Timespent spreadsheet

 * For the avoidance of doubt: fill out the sheet week number for the previous week


 * spreadsheet for week 39 (9/23): https://docs.google.com/spreadsheets/d/1UEmjIbv6voTF5bOnHdjOltw47AwPDtnmUo0h78wgYDY/edit?pli=1#gid=131825410

November Offsite/Summit

 * TechConf19: November 12–15
 * Offsite: November 16–20
 * Arrive the 15th
 * 16th is our fun day/outing
 * 17–20 are working days
 * 21st fly home
 * https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning

Book club

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
 * 2019-10-15
 * Read the blog post:  http://blog.codepipes.com/testing/software-testing-antipatterns.html
 * Let's discuss! https://etherpad.wikimedia.org/p/releng-book-club

Monthly reflection on accomplishments - September '19 edition

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
 * Add as you have them!


 * Scap 3.12.1-1 released/deployed
 * Refactored Zuul layout to use per-branch pipelines
 * Lets you run arbitrary code against a working MediaWiki install
 * The phabricator "Report Error Code" form (https://phabricator.wikimedia.org/maniphest/task/edit/form/46/ ) has been updated with separate fields for the stack trace and error code/request id.
 * T232608 Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days
 * Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ (first step towards a saner config)
 * MediaWiki PHP support target modernised from 7.0+ to 7.2+ for 1.34 onwards. https://phabricator.wikimedia.org/T228342
 * Quibble 0.0.35 release
 * 1.34.0-wmf.24 branch cut was done /mostly/ with branch.py instead of make-wmf-branch.php (some small bugs remain to work out but it's very close)
 * Creating accounts was broken on beta cluster since 2019-09-08. It was fixed today (2019-09-25). https://phabricator.wikimedia.org/T232796
 * Phatality extension for Kibana deployed to production and used for reporting production errors into Phabricator.
 * Train blocker tasks created for 1.35.0-wmf.1-1.35.0-wmf.25
 * MediaWiki dev images are now automatically created as part of postmerge via the pipeline

Incoming/Needs attention

 * Pipeline: https://phabricator.wikimedia.org/T233316#5536374
 * CPT secrets: https://phabricator.wikimedia.org/T233092

New Month, New Workboard!

 * Please move from 201909 -> 201910 (and move up to TODO/close invalidate/whatever liberally)
 * 201909 https://phabricator.wikimedia.org/project/view/4253/
 * 201910 https://phabricator.wikimedia.org/project/view/4311/
 * Let me know in 1:1s if you're stuck/have a question about how to treat a task

Proposed New Rotating Position

 * Clinic Duty!
 * Analogous to:
 * SRE Clinic Duty
 * CPT Clinic Duty
 * Proposed scope
 * Triage tasks on #together INBOX/Release-Engineering-Team-TODO INBOX/Release-Engineering-Team-TODO (201XXX) INBOX
 * Handle "hey releng"-type requests in #wikimedia-releng (@meatbags for brennen)
 * +2 common requests (add x@y.com to ci whitelist)

Incoming from last week (aka this morning)

 * The Growth team is planning deployments for the newcomer tasks projects and we need to know November black out dates
 * CPT: Create a way to share a secret between MediaWiki and the testing framework in CI T233092
 * Kartotherian folks: T233316
 * Not blocked, but nice-to-have for Fundraising tech: php7.0 docker image for CRM tests  T230446
 * Fundraising emails to English-speaking countries started going out this week - that traffic goes through donate.wikimedia.org (on the main cluster) before it hits payments-wiki (on the fr cluster), so let's nobody break it.

Outgoing next week

 * Blocked:
 * Blocking:
 * Updates:
 * Train Health
 * Last week: 1.34.0-wmf.25 - T220750
 * This week: 1.35.0-wmf.1 - T233849
 * Next week: 1.35.0-wmf.2 - T233850
 * This week: 1.35.0-wmf.1 - T233849
 * Next week: 1.35.0-wmf.2 - T233850

Train status and happenings

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Roles#Train_Conductor

OKRs, Goals, Betterworks, Oh my!

 * Concerns? Questions no one can answer?
 * Additionally (yes, that's right)...Team Goals
 * https://etherpad.wikimedia.org/p/RelEngTeam-Weekly_2019-2020-Q2-Goals

Standup!

 * Make sure your tasks are correct on our kanban board!
 * https://phabricator.wikimedia.org/project/view/4253/ - 2019-09
 * https://phabricator.wikimedia.org/project/view/4311/ - 2019-10

Antoine

 * What I planned vs did last week
 * Had Castor instance relocated to have more CPU https://phabricator.wikimedia.org/T232646 \o/
 * Extract Selenium to standalone job https://phabricator.wikimedia.org/T232759
 * Gerrit and java GC debugging (some bot was still hammering the master) https://phabricator.wikimedia.org/T234328
 * Found a slowness bug in php-xdebug from component/php72 ( https://phabricator.wikimedia.org/T234418 )
 * What I plan to do this week
 * More extraction of Selenium to standalone jobs ( https://phabricator.wikimedia.org/T232759 )
 * Teach Quibble to record duration of each stages and report at end of builds (partly done)
 * OKR?
 * Reviews for Quibble
 * Stretch VM are slower than the Jessie ones https://phabricator.wikimedia.org/T232706
 * What I'm blocked on
 * Status of HHVM in production? Which implies: can we drop HHVM from CI?
 * Other?
 * Some cloudvirt are unreasonably slow. Still going on
 * https://phabricator.wikimedia.org/T225713  "CPU scaling governor fix up"
 * https://phabricator.wikimedia.org/T223971 "old cloudvirt are CPU slow"
 * Logstash on beta has been broken for months https://phabricator.wikimedia.org/T233134
 * Logstash on beta has been broken for months https://phabricator.wikimedia.org/T233134

Brennen

 * What I planned vs did last week
 * ✅ generate MediaWiki + Apache dev image from pipeline (!!!)
 * https://gerrit.wikimedia.org/r/c/mediawiki/core/+/539589
 * https://gerrit.wikimedia.org/r/c/integration/config/+/539447
 * https://gerrit.wikimedia.org/r/c/mediawiki/core/+/525972
 * https://gerrit.wikimedia.org/r/c/releng/dev-images/+/525842
 * ❌ Send Lars my PGP key, after I figure out what's wrong with my PGP key
 * ✅ Read Lars's latest CI document
 * ✅ Final(?) CI WG meeting on Friday
 * ✅ Investigate whether https://phabricator.wikimedia.org/T229280 is really still happening
 * Focus on dev environment questions
 * Unplanned: Reviewed some pipelinelib patches
 * What I plan to do this week
 * Dev environment experimentation
 * Pair with Jeena on generating some service dev images in pipeline
 * T234391 Point deployment-charts/mediawiki-dev at latest dev image published by pipeline
 * T234379 MediaWiki pipeline config: Correctly tag development images with dev
 * Put OKRs in Betterworks
 * Read for book club
 * Figure out what violence Enigmail and Keybase did to my GPG config
 * What I'm blocked on
 * Other?
 * Other?

Dan

 * What I planned vs did last week
 * Seakeeper K8s Proposal
 * [yay!]++
 * What I plan to do this week
 * Deployment Pipeline fails with CPS error for Kartotherian
 * Branch REL1_34 for MediaWiki and deployed extensions
 * 1.34.0-wmf.25 deployment blockers
 * Seakeeper K8s Proposal
 * What I'm blocked on
 * Other?
 * Other?

Greg

 * What I planned vs did last week
 * ✅ Tech-mgt F2F, meet Grant, new CTO
 * ✅ Leveling...
 * What I plan to do this week
 * TechConf session drafting
 * OKR drafting, aligning
 * Goal setting cross EngProd
 * What I'm blocked on
 * The announcement for EngProd creation :) (just waiting, not actually blocking anything other than nice feelies)
 * Other?

James

 * What I planned vs did last week
 * Vacation all week – I worked on Thursday anyway, because I was bored.
 * What I plan to do this week
 * Supporting 1.34 release branch cut/etc.
 * OKR fun.
 * HHVM removal from CI and later MW
 * Static configuration progress https://phabricator.wikimedia.org/T223602
 * Iterating from CodeReview metrics/KPI proposal to make some for CI
 * What I'm blocked on
 * Seeking input from deployers/configurators on plans for static configuration changes
 * Other?

Jean-Rene

 * What I planned vs did last week
 * vacation
 * What I plan to do this week
 * vacation
 * What I'm blocked on
 * Other?
 * Other?

Jeena

 * What I planned vs did last week
 * ✅ Update parsoid blubberfile to new version
 * ✅ local-charts meeting
 * ❌ Get parsoid image built in CI. Same for restbase?
 * ✅ Fix some issues with mediawiki chart
 * ✅ Fix some issues with parsoid chart
 * bring up issue with deployment-charts about templates with same name
 * ✅ Add ideas to local dev brainstorm
 * ✅ local dev meeting
 * ✅ docker sig meeting?
 * What I plan to do this week
 * make example of deployment charts template problem for Alexandros
 * OKR writing
 * local dev meeting & prep
 * Work with Brennen to get parsoid image built in CI. Same for restbase? Good question.
 * Explore what's needed for new php parsoid service in terms of local-charts/deployment-charts
 * Reading for book club
 * What I'm blocked on
 * Other?
 * Other?

Lars

 * What I planned vs did last week
 * ❌ Get feedback on v3 of CI arch doc. Process it.
 * ✅ Read up on Zuul dependent pipelines.
 * ✅ With CI WG, decide on CI engine for future CI. Make plan on what next.
 * ❌ Update engprod list on PGP keysigning at offsite.
 * ❌ Read the book club article. Let subconsicous start work on it.
 * ✅ filled in my OKRs in betterworks
 * What I plan to do this week
 * Seek feedback on v3 of CI arch doc.
 * Set up Argo in a K8s cluster somewhere.
 * Update engprod list on PGP keysigning at offsite.
 * Read the book club article. Let subconsicous start work on it.
 * What I'm blocked on
 * K8s - could do with some help
 * Other?

Mukunda

 * What I planned vs did last week
 * ✅ Train
 * ✅ Get familiar with betterworks
 * ✅ Deploy phatality to production
 * ✅ Develop some OKRs
 * What I plan to do this week
 * Branch REL1_34 with Dan
 * Work on branch.py
 * Phabricator Deploy
 * Work on getting phabricator to notify you when attempting to submit a duplicate phatality report.
 * What I'm blocked on
 * Other?
 * Other?

Tyler

 * What I planned vs did last week
 * ✅ tech-mgr f2f
 * What I plan to do this week
 * Goals
 * Meetings
 * Gerrit1001
 * What I'm blocked on
 * Other?
 * Other?

Zeljko

 * What I planned vs did last week
 * ✅ T232598 Update existing Selenium documentation
 * ❌ T232638 Add accessibility check for Homepage (GrowthExperiments)
 * What I plan to do this week
 * T213268 Upgrade webdriverio to version 5 in mediawiki/core
 * T234002 Make MediaWiki Wdio tests less slow (Sept 2019)
 * T199113 All repositories with Selenium tests should use wdio-mediawiki
 * T234311 Fix and restore daily browser tests for TwoColConflict
 * T232638 Add accessibility check for Homepage
 * What I'm blocked on
 * T233960 Error: error installing: the server could not find the requested resource
 * SoS on 2019-10-09 is at the same time as Tech All Hands
 * Other?
 * Should I be working on individual goals? apparently not
 * October 1 was 7 (seven!) years I've been working for WMF