Wikimedia Release Engineering Team/Checkin archive/20191023
2019-10-23[edit]
Vacations/Important dates[edit]
- October 8: Željko holiday (Independence Day)
- October 11: Željko conference (https://2019.webcampzg.org/ )
- October 14 (Indigenous Peoples' Day): US Staff
- October 24–27: James at WikidataCon 2019 in Berlin
- November 1 (All Saints' Day): Željko holiday
- November 8–10: WikiCon North America (James)
- November 11 (Veterans' Day): US Staff
- November 12–15: TechConf19
- November 16–20: Offsite
- November 21-30: Jeena
- November 25-29: Dan
- November 28–29 (Thanksgiving): US Staff
- December 1-2: Jeena
- December 6: Lars (Finnish Independence Day)
- December 25–31 (Christmas): US Staff
- December 25–26: Lars, Željko, Christmas
- 1 January (New Year's Day): US Staff, Lars, Željko
- 20 January (Martin Luther King Jr. Day): US Staff
Rotating positions[edit]
Train[edit]
- Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R
- 14 Oct - wmf.2 - Jeena + Lars
- 21 Oct - wmf.3 - Lars + Brennen
- 28 Oct - wmf.4 - Brennen + Mukunda
- 4 Nov - wmf.5 - Mukunda + Antoine
- 11 Nov - wmf.6 - No Train (Tech conf)
- 18 Nov - wmf.7 - No Train (Off site)
- 25 Nov - wmf.8 - Antoine + Jeena (Thanksgiving)
- 2 Dec - wmf.9 - TBD
- 9 Dec - wmf.10 - TBD
- 16 Dec - wmf.11 - TBD
- 23 Dec - wmf.12 - No Train or any other deploys (Winterval)
- 30 Dec - wmf.13 - No Train or any other deploys (Still Winterval)
- 6 Jan - wmf.14 - TBD
- 13 Jan - wmf.15 - TBD
SoS[edit]
- Aug 14 onward: Zeljko 🎸 🎷 \o/
Team Business[edit]
Timespent spreadsheet[edit]
- For the avoidance of doubt: fill out the sheet week number for the previous week
- spreadsheet for week 42 (2019-10-14):https://docs.google.com/spreadsheets/d/1YS_2eUQWUOnygshMfxgGPu93GKGCO5t00yjUREQ5B-I/edit#gid=442607278
- reminder: your time spent doesn't need to total to 100%
Clinic Duty Proposal[edit]
November Offsite/Summit[edit]
- TechConf19: November 12–15
- Offsite: November 16–20
- Arrive the 15th
- 16th is our fun day/outing
- 17–20 are working days
- 21st fly home
- https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
- Dietary restrictions due TODAY!
- New proposal on etherpad: how does EngProd want to use betterworks
- Outing day! Vote please
Book club[edit]
- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
- Good discussion! https://etherpad.wikimedia.org/p/releng-book-club
Monthly reflection on accomplishments - October '19 edition[edit]
- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
- Add as you have them!
October[edit]
- Dev images are now automatically created as part of postmerge via the pipeline for:
- Parsoid
- Soon: RestBASE
- Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
- Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
- Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
- Quibble 0.0.38 & 0.0.39 released for mediawiki/tools/api-testing
- Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
- HHVM removed from CI and MediaWiki.
- Gerrit is on gerrit1001 now
- … and so is most of the code review. ;-)
Incoming/Needs attention[edit]
Scrum of Scrums for next week[edit]
Internal team notes, not for pasting into the SoS notes[edit]
Incoming from last week (aka this morning)[edit]
Outgoing next week[edit]
- Blocked:
- Site Reliability Engineering: logstash-beta.wmflabs.org does not receive any mediawiki events https://phabricator.wikimedia.org/T233134
- Updates:
- v3 of CI arch document for new CI wants review and feedback https://www.mediawiki.org/wiki/User:LarsWirzenius/NewCI
- Gerrit is now on a newer (larger) server
- There was some problem with rollout (patches looking unmerged that were actually merged)
- Resolved now
- All gory details on https://phabricator.wikimedia.org/T236114
- Train Health
- Last week: 1.35.0-wmf.2 - phab:T233850
- This week: 1.35.0-wmf.3 - phab:T233851
- Next week: 1.35.0-wmf.4 - phab:T233852
Callouts[edit]
Train status and happenings[edit]
Standup![edit]
- Make sure your tasks are correct on our kanban board!
Antoine[edit]
Last week[edit]
- Best thing
- Paired git fixage with Tyler for hmm lot of hours. Pair programming bash over Google Meet is a great experience.
- For Wikibase.git : extracted Selenium to a standalone jobs ( https://phabricator.wikimedia.org/T232759 )
- Quibble 0.0.39 for mediawiki/tools/api-testing (very cool project)
- Worst thing
- What I planned vs did
- Switch integration/config to use jenkins-job-builder from pypi instead of our git repo
- PostBuildScript jenkins plugin upgrade prepared
- Nerd snipped on some weird logstash index fields limit https://phabricator.wikimedia.org/T234564
- Unplannned
- Gerrit post migration magic (thank you Tyler!) (thcipriani: thank you!)
This week[edit]
- What I plan to do
- TechConf preparation with Kosta / WMDE-Fisch
- Blockers
- WMCS blocked: they need CI for a golang projects https://phabricator.wikimedia.org/T236203 (dduvall: we have a docker-pkg-built golang base image; see blubber's .pipeline/blubber.yaml)
- php-xdebug needs upgrade (slows down phpunit coverage report)
Other[edit]
Brennen[edit]
Last week[edit]
- Best thing
- Pairing in general, messing with sshecret.
- Worst thing
- Moments of panic about logs prior to deploy.
- What I planned vs did
Done Make sure local-charts install script works on buster
Done Further TechConf input, CI session planning
In progress Train pairing
- Unplannned
- Some time investigating broken mediawiki/core dev images
- Some time investigating toollabs to restart dockerregistry
This week[edit]
- What I plan to do
- Train
- Sort out all hands travel
- Dev environment hackery
- Check out k3sup?
- Think about TechConf local dev sessions / discuss
- Improve sshecret pull req: https://github.com/thcipriani/sshecret/pull/2
- Blockers
Other[edit]
Dan[edit]
Last week[edit]
- Best thing
- Diagramming Seakeeper subsystems
- Worst thing
- Fixing janky Jenkins reports
- starting in August, 30-day reports should be accurate
- Fixing janky Jenkins reports
- What I planned vs did
- Seakeeper proposal, fleshed out descriptions of CI subsystems for proposed k8s namespaces
- Unplannned
- Fixed problems with Jenkins reports at https://people.wikimedia.org/~dduvall/jenkins
- Build retention differences made for inaccurate 30-day reports. Working around this by aggregating from past build archives.
- Fixed problems with Jenkins reports at https://people.wikimedia.org/~dduvall/jenkins
This week[edit]
- What I plan to do
- Seakeeper proposal, gathering capacity/resource utilization stats
- Blockers
Other[edit]
Greg[edit]
Last week[edit]
- Best thing
- The TechConf program committee is great to work with, even when we are stressed
- Worst thing
- Time constraint on the Tuning Session slide creation (which had knock off effects of figuring out what the hell these OKRs and metrics really are)
- Effing moving
- What I planned vs did
- techconf
- tuning session
- Unplannned
This week[edit]
- What I plan to do
- tech conf schedule creation (well, version 2.0 of the schedule) and post it
- engprod logistics setup
- Blockers
- time
Other[edit]
- TechConf office hours happening next week on IRC, email going out shortly announcing them
- Desire/stretch: create a homepage for Engineering Producitivity somewhere :)
James[edit]
Last week[edit]
- Best thing
- Made progress discussing with Timo way forward for static configuration work; next step is autogenerating dblists from YAML.
- Worst thing
- Real jetlag for the first time in a few years. Meh.
- What I planned vs did
Done OOUI migration to node 10 / PHP72
Not done Fundraising Tech CRM image to PHP70 (could do with input from Antoine) https://gerrit.wikimedia.org/r/c/integration/config/+/540683 for https://phabricator.wikimedia.org/T230446
Partially done More static configuration work, a bit delayed.
Done More TechConf thinking/planning.
Done UBNs as they arise.
- Unplannned
- Renamed "unversioned" PHP (PHP70) images to be explicit
- Antoine: THANK YOU
- Helping Analytics put a "stats for this site" link in the site footer T235803
- Renamed "unversioned" PHP (PHP70) images to be explicit
This week[edit]
- What I plan to do
- WikiDataCon from tomorrow onwards; back in the US next Wednesday.
- Whatever breaks.
- YAML/static builds of dblists.
- Blockers
- —
Other[edit]
- —
Jean-Rene[edit]
Last week[edit]
- Best thing
- Worst thing
- What I planned vs did
- Unplannned
This week[edit]
- What I plan to do
- Blockers
Other[edit]
Jeena[edit]
Last week[edit]
- Best thing
- finished restrouter chart...hopefully \o/ \O/ \:D/
- Worst thing
- spent time trying to debug Visual Editor not working, unexpectedly there was an error in the browser console
- What I planned vs did
Not done timespent spreadsheet
Not done work with zeljko on getting local-charts running - Željko: feel free to pospond until you're done with train
In progress look at seakeeper proposal
Done train
Done update restrouter deployment-charts config for dev purposes
- Unplannned
- debug broken mw core image
This week[edit]
- What I plan to do
- tech conf session planning
- changes to mw core image publish pipeline to log more and fail if prereq script errors
- re-open patchsets to build restbase dev image
- timespent spreadsheet
- work with zeljko on getting local-charts running
- ...
- Blockers
Other[edit]
- tiny sunflower
Lars[edit]
Last week[edit]
- Best thing
- Going to TechConf
- Worst thing
- Not going to a mountain in TN
- What I planned vs did
Done Help Jeena with train, a little, as backup.
Done Seek feedback on v3 of CI arch doc.
Done Update engprod on PGP signing for offsite.
Not done Set up Argo in a K8s cluster somewhere.
- Needs uninterrupted time, and I've been easily distractable. Also train.
- Unplannned
- Was invited to TechConf, and did much arranging of travel.
- More review of Dan's Seekeeper proposal.
- Wrote up book club discussion on wiki.
- Failed to deploy train, didn't get to group0 even, but much excitement was had by everyone.
- Wrote a wrapper around rsync, in case it's useful some day.
- Registered for All Hands, started arranging travel.
This week[edit]
- What I plan to do
- Train, 1.35.0-wmf.3
- Skim Go book, write a small program from scratch: fetch a web page, diff against previously fetched version.
- Attempt to set up study groups, or possibly SIGs, for Go and Rust.
- Blockers
Other[edit]
- The Darkness is here until about March or April, expect SADness
- performance has TT; releng has BB, DD, and MM; qte has no doubled initials
- :-)
- EngProd has GG :P
- My "legal" name is Tyler Henry Turley-Cipriani, so TTish
Mukunda[edit]
Last week[edit]
- Best thing
- Phab1001 is up and running
- Worst thing
- Didn't get around to testing the "Ferret" Fulltext search engine in phab (yet)
- What I planned vs did
- Mostly
Done Finish getting branch.py ready for the wmf production branching (WIP: https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/release/+/543248/ )
Done Try to build some tests for branch.py
Done More work on phabricator high-availability & disaster recovery planning.
- phab1001 should be nearly ready to become the primary server now.
Not done Test out phabricator's built-in search engine with an eye toward dropping elasticsearch if the integrated search performs as well.
- Mostly
- Unplannned
This week[edit]
- What I plan to do
- Test out phabricator's built-in search engine
- Drum up interest in patch-triage (email wikitech-l, not sure what else to do)
- Get branch.py ready for full automation in releases-jenkins
- Deploy phatality (after https://gerrit.wikimedia.org/r/#/c/545593/ merges)
- Deploy phabricator
- Blockers
- code review of https://gerrit.wikimedia.org/r/#/c/545593/
Other[edit]
Tyler[edit]
Last week[edit]
- Best thing
- Seeing the train pairing thing work :)
- pairing with Antoine on Gerrit migration fallout
- Worst thing
- Gerrit migration fallout
- What I planned vs did
Done connect with SRE re New CI (again)
Done Reach out to folks about being on Sr. SWE interview loop
Done announce dec code freeze
Done? tuning session prep
Done Gerrit -> gerrit1001
- Unplannned
- Gerrit migration fallout
- Updates to deployment-calendar scheduler -- awight noticed some issues
This week[edit]
- What I plan to do
- Resumé review
- Work out clinic duty rotation
- Next bookclub email
- Gerrit/JVM tuning to make use of Moar Memory™
- Blockers
- Time
- Dog
- Blazey at conference
Other[edit]
- Picking up Blazey at airport this evening, may cut out a bit early
Zeljko[edit]
Last week[edit]
- Best thing
- Email from Safeguard: "Please note: all documentation must be password protected from this point using the following (password)"
- Worst thing
- T236282 `npm run selenium` fails with `SyntaxError: Unexpected identifier`
- What I planned vs did
In progress T234610 ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73
In progress T199113 All repositories with Selenium tests should use wdio-mediawiki
- Unplannned
- Safeguard
- T235795 Create test for GrowthExperiments to check for ability to disable homepage
- T233820 mediawiki/core Selenium tests fail when targeting mediawiki/vagrant VM
- T236282 `npm run selenium` fails with `SyntaxError: Unexpected identifier`
This week[edit]
- What I plan to do
- T234610 ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73
- T199113 All repositories with Selenium tests should use wdio-mediawiki
- Blockers
Other[edit]
- Saw Rambo: Last Blood. Not recommended.