Wikimedia Release Engineering Team/Checkin archive/20191030

= 2019-10-30 =

Vacations/Important dates

 * https://office.wikimedia.org/wiki/HR_Corner/Holiday_List
 * How to do it


 * November 1 (All Saints' Day): Željko, Antoine holiday
 * November 8–10: WikiCon North America (James)
 * November 11 (Veterans' Day): US Staff
 * November 12–15: TechConf19
 * November 16–20: Offsite
 * November 21–30: Jeena
 * November 25–29: Dan
 * November 28–29 (Thanksgiving): US Staff


 * December 1–2: Jeena
 * December 6: Lars (Finnish Independence Day)
 * December 25–31 (Christmas): US Staff (+ everyone?)
 * December 25–26: Lars, Željko, Christmas


 * 1 January (New Year's Day): US Staff, Lars, Željko
 * 20 January (Martin Luther King Jr. Day): US Staff

Train

 * Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R


 * 14 Oct - wmf.2 - Jeena + Lars
 * 21 Oct - wmf.3 - Lars + Brennen


 * 28 Oct - wmf.4 - Brennen + Mukunda
 * 4 Nov - wmf.5 - Mukunda + Antoine
 * 11 Nov - wmf.6 - No Train (Tech conf)
 * 18 Nov - wmf.7 - No Train (Off site)
 * 25 Nov - wmf.8 - Group0 on Tuesday Brennen + Mukunda (Thanksgiving)
 * 2 Dec - wmf.8 part 2 - Wednesday Group1 + Thursday Group2
 * 9 Dec - wmf.10 - TBD
 * 16 Dec - wmf.11 - TBD
 * 23 Dec - wmf.12 - No Train or any other deploys (Winterval)
 * 30 Dec - wmf.13 - No Train or any other deploys (Still Winterval)
 * 6 Jan - wmf.14 - TBD
 * 13 Jan - wmf.15 - TBD

SoS

 * Aug 14 onward: Zeljko 🎸 🎷 \o/

Timespent spreadsheet

 * For the avoidance of doubt: fill out the sheet week number for the previous week


 * spreadsheet for week 43 (2019-10-21): https://docs.google.com/spreadsheets/d/1YS_2eUQWUOnygshMfxgGPu93GKGCO5t00yjUREQ5B-I/edit#gid=337909193
 * reminder: your time spent doesn't need to total to 100%

Clinic Duty Proposal

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Clinic_Duty
 * ❌ thcipriani JFDI®

November Offsite/Summit

 * TechConf19: November 12–15
 * Offsite: November 16–20
 * Arrive the 15th
 * 16th is our fun day/outing
 * 17–20 are working days
 * 21st fly home
 * https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
 * Outing day! Vote please

Book club

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
 * Check your email

Monthly reflection on accomplishments - October '19 edition

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
 * Add as you have them!

October

 * Dev images are now automatically created as part of postmerge via the pipeline for:
 * Parsoid
 * Soon: RestBASE
 * (different from RESTbase? ;-))
 * Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
 * Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
 * Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
 * Quibble 0.0.38 & 0.0.39 released for mediawiki/tools/api-testing
 * Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
 * HHVM removed from CI and MediaWiki.
 * Gerrit is on gerrit1001 now
 * … and so is most of the code review. ;-) :)
 * Unforked Jenkins Job Builder

Incoming/Needs attention

 * Wikidata load issues; NFA for us right now. T236928 // T234948 // …
 * Branch cut didn't work with the new script, had to use the old one.

Outgoing next week

 * Blocked:
 * Site Reliability Engineering: logstash-beta.wmflabs.org does not receive any mediawiki events https://phabricator.wikimedia.org/T233134
 * [Low priority] Fundraising Tech: Code review on https://gerrit.wikimedia.org/r/544071 appreciated to unblock https://phabricator.wikimedia.org/T192167
 * Updates:
 * v3 of architecture document for new continuous integration wants review and feedback https://www.mediawiki.org/wiki/User:LarsWirzenius/NewCI
 * Train Health
 * Last week: 1.35.0-wmf.3 - T233851
 * This week: 1.35.0-wmf.4 - T233852
 * Next week: 1.35.0-wmf.5 - T233853 - last train before two week break

Callouts

 * Release Engineering
 * Next week train last train (1.35.0-wmf.5 - T233853) is the last train before two week break
 * v3 of architecture document for new continuous integration wants review and feedback https://www.mediawiki.org/wiki/User:LarsWirzenius/NewCI

Train status and happenings

 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Roles#Train_Conductor


 * No Gerrit GC cycle during branch cut. How was branch cut?

Standup!

 * Make sure your tasks are correct on our kanban board!
 * https://phabricator.wikimedia.org/project/view/4311/ - 2019-10

Last week

 * Best thing
 * JJB to be used directly from integration/config: tox -e jenkins-jobs --
 * Will probably phase out integration/jenkins-job-builder
 * Couple of JJB patch upstream to speed it up (from 28s down to 4s on my machine)
 * Fixed Zuul gearman monitoring probe. Raised threshold
 * sync-up Slides for tech conf
 * Worst thing
 * Docker containers on Stretch are 20% slower than Jessie ones T236675
 * liw: could this be a spectre problem?
 * hashar: I think there is a problem with that security patch and docker
 * What I planned vs did
 * Unplannned
 * Unplannned
 * Unplannned

This week

 * What I plan to do
 * Blockers
 * Blockers

Other

 * Not there friday Nov 1st (holiday: All Saints)

Last week

 * Best thing
 * Decreasing sense of terror about train.
 * Worst thing
 * Train.
 * What I planned vs did
 * ✅ Train backup for Lars / train lead this week
 * ✅ Sort out all hands travel
 * Dev environment hackery
 * ❌ Check out k3sup?
 * ✅ Think about TechConf local dev sessions / discuss
 * And also
 * ❌ Improve sshecret pull req: https://github.com/thcipriani/sshecret/pull/2
 * Decided after discussion that current state is Good Enough™, Tyler merged
 * May devote some future yak shaving time
 * Unplannned
 * T236333: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker
 * Discovered Antoine's jjb changes firsthand
 * Help on some local-charts bugfixen:
 * T236471: `Error: validation failed` during `make deploy`
 * T233960: Error: error installing: the server could not find the requested resource
 * Bumped helm to 2.15.1

This week

 * What I plan to do
 * Train, including some updates to docs
 * Improve a log-tailing script and document
 * Write a "This Week in Logspam"
 * Work on tech conf session
 * Review pending local-charts patches
 * Take a crack at TimedMediaHandler on local-charts
 * Blockers

Last week

 * Best thing
 * Learned a lot about statistical forecasting
 * Worst thing
 * Learned a lot about statistical forecasting
 * What I planned vs did
 * Finished (enough) the Seakeeper proposal for presentation to SRE Service Ops
 * https://docs.google.com/document/d/1b6sqmfdcH4XL8wayL5OOaJX9xOyDq-osVEEkzzIWWY4/edit?userstoinvite=akosiaris%40wikimedia.org&ts=5db97664#
 * Presented proposal to SRE Service Ops
 * Unplannned
 * Fixed up Jenkins build reports

This week

 * What I plan to do
 * Following conversations about Zuul's Dependent Pipeline Manager and how it will need to fit with New CI, looking into it a bit
 * Calculating overhead
 * Looking at implementation
 * Revisiting notes from Dec 2018 offsite and how we might test it's efficacy
 * Start working with Lars on pipelinelib user documentation
 * Blockers

Last week

 * Best thing
 * I have a desk!
 * Worst thing
 * Thought I lost the hardware for the desk for a day, luckily my partner is smarter than I
 * What I planned vs did
 * TechConf: mostly on track, the final countdown begins ( https://www.youtube.com/watch?v=9jK-NcRmVcw )
 * Unplannned
 * Things

This week

 * What I plan to do
 * TechConf!
 * Offsite
 * Blockers

Last week

 * Best thing
 * WikidataCon.
 * Worst thing
 * WikidataCon-related travel.
 * What I planned vs did
 * ✅ WikiDataCon from tomorrow onwards; back in the US next Wednesday.
 * ❌ YAML/static builds of dblists; not had the time yet today to deploy the next step.
 * Unplannned
 * Some extra PHP 7.0 / 7.1 CI removal patches for MW-related libraries.
 * Additional linting: we now lint require JS calls to work with ResourceLoader https://github.com/wikimedia/eslint-plugin-mediawiki
 * Some more clean-up/support for PHPUnit upgrade, Phan upgrade, removal of globals, etc. T192167, T159283, T160811, …
 * Built some early images for PHP74 CI testing; paused waiting for PHP 7.4.0 release. T236132
 * Explored replacing Travis CI with GitHub Actions CI for some of our GitHub-hosted repos. https://github.com/wikimedia/less.php/pull/18
 * Helped Analytics by putting a "Statistics" link in the footer of every page: T235803.

This week

 * What I plan to do
 * Static YAML generation of dblists
 * Static YAML generation of InitialiseSettings
 * LocalDev support
 * TechConf planning
 * Blockers
 * Waiting on feedback from FR-Tech on whether we can proceed with PHPUnit stuff as it effects them (pinged in SoS) T192167

Other

 * Working from the East Coast from Sunday onwards.

Last week

 * Best thing
 * Worst thing
 * What I planned vs did
 * Unplannned
 * What I planned vs did
 * Unplannned
 * Unplannned

This week

 * What I plan to do
 * Blockers
 * Blockers

Last week

 * Best thing
 * fixed some bugs in local-charts
 * Worst thing
 * planning is hard
 * What I planned vs did
 * tech conf session planning
 * ❌ changes to mw core image publish pipeline to log more and fail if prereq script errors
 * ❌ re-open patchsets to build restbase dev image
 * ❌ timespent spreadsheet
 * work with zeljko on getting local-charts running
 * Unplanned
 * code review for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/544629

This week

 * What I plan to do
 * tech conf session planning
 * changes to mw core image publish pipeline to log more and fail if prereq script errors
 * re-open patchsets to build restbase dev image
 * timespent spreadsheet
 * fix admin username/password issue for mw core on local-charts
 * Blockers
 * need Marko to review my deployment-charts restrouter patch (thcipriani: IIRC he's OOO today....)

Last week

 * Best thing
 * Ran the train and didn't break anything too badly
 * Worst thing
 * Realized that my technical writing isn't very useful
 * What I planned vs did
 * ✅ Train, 1.35.0-wmf.3
 * Skim Go book, write a small program from scratch: fetch a web page, diff against previously fetched version.
 * ✅ Attempt to set up study groups, or possibly SIGs, for Go and Rust.
 * Unplanned
 * First meeting with SRE on hardware to new CI and general sync on this.
 * First meeting in rebooted series of Deployment Pipeline meetings.

This week

 * What I plan to do
 * Ponder what came out of the SRE meeting on CI hardware meeting, and what I can learn from that, and what I can do for future discussssions on this.
 * Help prepare CI/CD related sessions at Tech Conf.
 * Respond to email thread started by Tyler. Especially: suggest goals and desired outcomes of the session and other related discussions from Tech Conf.
 * Fill out https://phabricator.wikimedia.org/T234641 best I can.
 * Sketch a distillation of my CI architecture document.
 * Update and fill out my personal travel checklist for Tech Conf.
 * Read up about last year's Tech Conf to be better prepared for this years'.
 * Install at least a rudimentary, "hello world" level Argo some K8s somewhere.
 * Skim Go book, write a small program from scratch: fetch a web page, diff against previously fetched version.
 * Have first Go study club meeting, on coming Monday. https://office.wikimedia.org/wiki/Go
 * Stretch: Arrange first meeting for Rust study club. https://office.wikimedia.org/wiki/Rust
 * Stretch: Look at Zuul's dependent pipeline manager and think hard about it for new CI.
 * Blockers

Other

 * Train conductor hat discussion was fun, again. Or is it train engineer hats?
 * http://monica-anderson.com/autumnsong.html

Last week

 * Best thing
 * Making good progress on phabricator redundancy and failover stuff with dzahn. phd (async job queue) is now running on two servers.
 * Worst thing
 * All of my planned tasks are ❌ because of a lot of time sunk on unplanned stuff.
 * What I planned vs did
 * ❌ Test out phabricator's built-in search engin
 * ❌ Drum up interest in patch-triage (email wikitech-l, not sure what else to do)
 * Working on it...
 * Get branch.py ready for full automation in releases-jenkins
 * still blocked on code review
 * ❌ Deploy phatality (after https://gerrit.wikimedia.org/r/#/c/545593/ merges)
 * Doing this today
 * ❌ Deploy phabricator
 * Tried with scap, it didn't work. Turns out it was my fault (scap.cfg change fixed it)
 * Unplanned
 * Phabricator dump is broken, so far I'm stumped: https://phabricator.wikimedia.org/T236507
 * Spent a bunch of time helping volker_e and converting desig/style-guide to scap: https://phabricator.wikimedia.org/T235677

This week

 * What I plan to do
 * Still working on branch.py, it didn't quite work as expected during the train branch cut this week, hope to have it working next time.
 * Deploy phabricator (with scap)
 * Deploy Phatality
 * Get branch.py working
 * Be the backup train conductor
 * Blockers
 * https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/543248

Last week

 * Best thing
 * New CI moving/public/momentum!
 * Worst thing
 * 113 Résumés + cover letters
 * What I planned vs did
 * ✅ Resumé review
 * ❌ Work out clinic duty rotation
 * ✅ Next bookclub email
 * ✅ Gerrit/JVM tuning to make use of Moar Memory™
 * Unplanned
 * Figured out Thanksgiving train (kinda/sorta)
 * Design moving to git-LFS

This week

 * What I plan to do
 * Work out clinic duty rotation
 * Techconf session planning
 * Gerrit review notes splitbrains
 * More gerrit tuning
 * Blockers

Other

 * It's cold https://www.pwsweather.com/obs/THCIPRIANI.html#

Last week

 * Best thing
 * local-charts powered mediawiki works on my machine!
 * Worst thing
 * /me grumbles something containing words: savings, daylight, time
 * What I planned vs did
 * ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 https://phabricator.wikimedia.org/T234610
 * All repositories with Selenium tests should use wdio-mediawiki https://phabricator.wikimedia.org/T199113
 * Unplanned
 * mediawiki-vagrant
 * MediaWiki-Vagrant should use the same Node.js version as Wikimedia production https://phabricator.wikimedia.org/T217113
 * `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) https://phabricator.wikimedia.org/T236282
 * local-charts
 * Can not log in to mediawiki https://phabricator.wikimedia.org/T236776
 * Error: error installing: the server could not find the requested resource https://phabricator.wikimedia.org/T233960
 * `Error: validation failed` during `make deploy` https://phabricator.wikimedia.org/T236471
 * `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` https://phabricator.wikimedia.org/T236393
 * `make install` fails with `“minikube-darwin-amd64” cannot be opened because the developer cannot be verified.` https://phabricator.wikimedia.org/T236402

This week

 * What I plan to do
 * Wikimedia Technical Conference 2019 Session: System level testing: patterns and anti-patterns with Selenium https://phabricator.wikimedia.org/T234635
 * Release new version of wdio-mediawiki https://phabricator.wikimedia.org/T236927
 * All repositories with Selenium tests should use wdio-mediawiki https://phabricator.wikimedia.org/T199113
 * T236620 Refactor GrowthExperiments Homepage Selenium test https://phabricator.wikimedia.org/T236620
 * Blockers

Other

 * [QA] Proposal to close this list https://lists.wikimedia.org/pipermail/qa/2019-October/002744.html
 * Jeena, Brennen (local-charts)
 * Can not log in to mediawiki https://phabricator.wikimedia.org/T236776
 * Capitalize MediaWiki user (from admin to Admin) https://gerrit.wikimedia.org/r/c/releng/local-charts/+/546988
 * TLDR section in README https://gerrit.wikimedia.org/r/c/releng/local-charts/+/546990
 * Antoine
 * ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 https://phabricator.wikimedia.org/T234610