Wikimedia Release Engineering Team/Checkin archive/20191009
2019-10-09[edit]
Vacations/Important dates[edit]
- October 8: Željko holiday (Independence Day)
- October 11: Željko conference (https://2019.webcampzg.org/ )
- October 14 (Indigenous Peoples' Day): US Staff
- October 25–26: James at WikidataCon 2019 in Berlin
- November 11 (Veterans' Day): US Staff
- November 12–15: TechConf19
- November 16–20: Offsite
- November 21-30: Jeena
- November 25-29: Dan
- November 28–29 (Thanksgiving): US Staff
- December 1-2: Jeena
- December 6: Lars (Finnish Independence Day)
- December 25–31 (Christmas): US Staff
- December 25–26: Lars, Christmas
- 1 January (New Year's Day): US Staff, Lars
- 20 January (Martin Luther King Jr. Day): US Staff
Rotating positions[edit]
Train[edit]
- Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R
- Jun 24 - wmf.11 - Jeena (with Mukunda)
- Jul 1 - wmf.12 - No Train (Fourth of July)
- Jul 8 - wmf.13 - Jeena
- Jul 15 - wmf.14 - Lars (with Antoine)
- Jul 22 - wmf.15 - Lars
- Jul 29 - wmf.16 - Brennen (with Tyler)
- Aug 5 - wmf.17 - Brennen
- Aug 12 - wmf.18 - No Train (Wikimania)
- Aug 19 - wmf.19 - Zeljko 😱
- Aug 26 - wmf.20 - Zeljko 😭
- Sep 2 - wmf.21 - Antoine
- Sep 9 - wmf.22 - Antoine
- Sep 16 - wmf.23 - Mukunda
- Sep 23 - wmf.24 - Mukunda
- Sep 30 - wmf.25 - Tyler Dan
- 7 Oct - 1.35.0-wmf.1 - Tyler Dan
- 17 Oct - wmf.2 - Jeena + Lars
- 21 Oct - wmf.3 - Lars + Brennen
- 28 Oct - wmf.4 - Brennen + Mukunda
- 4 Nov - wmf.5 - Mukunda + Antoine
- 11 Nov - wmf.6 - No Train (Tech conf)
- 18 Nov - wmf.7 - No Train (Off site)
- 25 Nov - wmf.8 - Antoine + Jeena (Thanksgiving)
- 2 Dec - wmf.9 -
- 9 Dec - wmf.10 -
- 16 Dec - wmf.11 -
- 23 Dec - wmf.12 - No Train (Winterval)
- 30 Dec - wmf.13 - No Train (Still Winterval)
- 6 Jan - wmf.14 -
- 13 Jan - wmf.15 -
TODO thcipriani to figure out thanksgiving wit hfolks
- antoine ok for thursday, can cut branch/sec patch on tuesday. Rest depend on jetlag of doom :-\
SoS[edit]
- Aug 14 onward: Zeljko
- SoS on 2019-10-09 is at the same time as Tech All Hands
Team Business[edit]
Timespent spreadsheet[edit]
- For the avoidance of doubt: fill out the sheet week number for the previous week
- spreadsheet for week 40 (2019-09-30): https://docs.google.com/spreadsheets/d/1YS_2eUQWUOnygshMfxgGPu93GKGCO5t00yjUREQ5B-I/edit#gid=0
- Ping Antoine, Brennen, Greg, Jeena, Lars
Clinic Duty Proposal[edit]
November Offsite/Summit[edit]
- TechConf19: November 12–15
- Offsite: November 16–20
- Arrive the 15th
- 16th is our fun day/outing
- 17–20 are working days
- 21st fly home
- https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
Book club[edit]
- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
- 2019-10-15
- Read the blog post: http://blog.codepipes.com/testing/software-testing-antipatterns.html
- Let's discuss! https://etherpad.wikimedia.org/p/releng-book-club
- 2019-10-15
Monthly reflection on accomplishments - September '19 edition[edit]
- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
- Add as you have them!
- Scap 3.12.1-1 released/deployed
- Refactored Zuul layout to use per-branch pipelines
quibble -c
Lets you run arbitrary code against a working MediaWiki install- The phabricator "Report Error Code" form (https://phabricator.wikimedia.org/maniphest/task/edit/form/46/ ) has been updated with separate fields for the stack trace and error code/request id.
- T232608 Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days
- Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ (first step towards a saner config)
- MediaWiki PHP support target modernised from 7.0+ to 7.2+ for 1.34 onwards. https://phabricator.wikimedia.org/T228342
- Quibble 0.0.35 release
- 1.34.0-wmf.24 branch cut was done /mostly/ with branch.py instead of make-wmf-branch.php (some small bugs remain to work out but it's very close)
- Creating accounts was broken on beta cluster since 2019-09-08. It was fixed today (2019-09-25). https://phabricator.wikimedia.org/T232796
- Phatality extension for Kibana deployed to production and used for reporting production errors into Phabricator.
- Train blocker tasks created for 1.35.0-wmf.1—1.35.0-wmf.25
- Dev images are now automatically created as part of postmerge via the pipeline for MediaWiki
September copied to https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments#September as of 2019-10-09 17:16 UTC
October[edit]
- Dev images are now automatically created as part of postmerge via the pipeline for:
- Parsoid
- Soon: RestBASE
- Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
- Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
- Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
- Quibble 0.0.38 released
- Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
- HHVM removed from CI and MediaWiki.
Incoming/Needs attention[edit]
Scrum of Scrums for next week[edit]
Internal team notes, not for pasting into the SoS notes[edit]
Incoming from last week (aka this morning)[edit]
Outgoing next week[edit]
- Blocked:
- Blocking:
- Updates:
- Train Health
- Last week: 1.35.0-wmf.1 - phab:T233849
- This week: 1.35.0-wmf.2 - phab:T233850
- Next week: 1.35.0-wmf.3 - phab:T233851
- Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
- Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
- Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
- Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
- Train Health
Callouts[edit]
Train status and happenings[edit]
OKRs, Goals, Betterworks, Oh my![edit]
- Concerns? Questions no one can answer?
- Additionally (yes, that's right)...Team Goals
Standup![edit]
- Make sure your tasks are correct on our kanban board!
Antoine[edit]
- What I planned vs did last week
- Tech conf sessions reviews. Gotta lead one or even two of them?
- Did Quibble reviews and a few faulty releases :-\
- Fall into a Gerrit trap
- Bitergia bot taking too much Gerrit resources, to be relocated to gerrit-replica (might have been done already)
- What I plan to do this week
- (LATE) More extraction of Selenium to standalone jobs ( https://phabricator.wikimedia.org/T232759 )
- Need full integration test for Quibble (namely actually run it on each patchset proposals)
- Stretch VM are slower than the Jessie ones https://phabricator.wikimedia.org/T232706
- Gotta check with WMCS team to benchmark that on their idling test cloudvirt hosts
- What I'm blocked on
- Some cloudvirt are too slow (T225713 / T223971), needs BIOS change.
- contint1001 has lot of dangling Docker images. Cleaned 47GBytes of them today (Oct 9th)
- For Gerrit we need a JMX Prometheus exporter to monitor the JVM https://phabricator.wikimedia.org/T184086
- Other?
- Logstash on beta is still broken https://phabricator.wikimedia.org/T233134
Brennen[edit]
- What I planned vs did last week
Done Pair with Jeena on generating some service dev images in pipeline
- parsoid's working now
- mediawiki:
Done T234379 MediaWiki pipeline config: Correctly tag development images with dev
- restbase should follow shortly (hopefully?)
Done Read for book club
In progress Dev environment experimentation
In progress T234391 Point deployment-charts/mediawiki-dev at latest dev image published by pipeline
Not done Put OKRs in Betterworks
Not done Figure out what violence Enigmail and Keybase did to my GPG config
- Unplanned:
- Feedback on job posting
- Mail to wikitech-l about dev image generation
- What I plan to do this week
- Merge deployment-charts/mediawiki-dev
- OKRs / Betterworks
- GPG
- Dev environment hackery
- Input/voting on tech conf sessions
- Familiarize myself with Seakeeper proposal
- What I'm blocked on
- Other?
- Closing on house tomorrow at 10:30am local, out for a bit
Dan[edit]
- What I planned vs did last week
Done 1.34.0-wmf.25 train
Done Branched REL1_34 🎉
- Ran into some issues here with the existing scripts and process
- Spent some time overhauling the scripts.
- Thanks for the help, Mukunda and James!
- Hopefully it will go more smoothly for the next sucker. 'tis one command.
In progress Worked on Seakeeper K8s Proposal
- What I plan to do this week
- 1.35.0-wmf.1 deployment blockers
- Deployment Pipeline fails with CPS error for Kartotherian
- Continue to flesh out Seakeeper K8s proposal
- Areas of specific focus to improve are: estimation of resource needs and delineation of access control (namespaces of subsytems, project workflows, and ideally who should have access)
- What I'm blocked on
- Other?
Greg[edit]
- What I planned vs did last week
- TechConf - mostly on track
- What I plan to do this week
- OKRs
- TechConf scheduling
- Please do review sessions and provide input
- Also, see the emails I sent to the releng@ list about these, seeking session leads
- Pipeline
- CI futures
- Airtable sense-making (probably will be back-burnered)
- What I'm blocked on
- Other?
James[edit]
- What I planned vs did last week
Done Supporting 1.34 release branch cut/etc.
Done OKR fun.
Done HHVM removal from CI and later MediaWiki
In progress Static configuration progress https://phabricator.wikimedia.org/T223602
In progress Iterating from CodeReview metrics/KPI proposal to make some for CI
- A bit of unplanned support for SDC.
- What I plan to do this week
- More static configuration progress
- PHP72 / PHPUnit8 work continues \o/
- 1.34.0-rc.0 support
- TechConf session scope/direction input
- UBNs as they arise.
- What I'm blocked on
- Input from Antoine on how to test https://gerrit.wikimedia.org/r/c/integration/config/+/539987 for https://phabricator.wikimedia.org/T234062
- Input from Antoine on how to fix https://gerrit.wikimedia.org/r/c/integration/config/+/540683 for https://phabricator.wikimedia.org/T230446
- (Antoine): I am late on my integration/config reviews (due to Gerrit/Quibble business)
- No worries. :-)
- (Antoine): I am late on my integration/config reviews (due to Gerrit/Quibble business)
- Still seeking input from deployers/configurators on plans for static configuration changes.
- Other?
Jean-Rene[edit]
- What I planned vs did last week
- What I plan to do this week
- What I'm blocked on
- Other?
Jeena[edit]
- What I planned vs did last week
Done make example of deployment charts template problem for Alexandros
In progress OKR writing
Done local dev meeting & prep
Done Work with Brennen to get parsoid image built in CI.
In progress Same for restbase? Good question.
Not done Explore what's needed for new php parsoid service in terms of local-charts/deployment-charts
Done Reading for book club
- mediawiki image benchmarking
- What I plan to do this week
- meetings
- timespent spreadsheet
- Explore what's needed for new php parsoid service in terms of local-charts/deployment-charts
- OKRs
- work with zeljko on getting local-charts running
- book club
- tech conf voting
- look at seakeeper proposal
- work on some other local dev poc
- What I'm blocked on
- Other?
Lars[edit]
- What I planned vs did last week
Not done Seek feedback on v3 of CI arch doc.
Not done Set up Argo in a K8s cluster somewhere.
Not done Update engprod list on PGP keysigning at offsite.
Not done Read the book club article. Let subconsicous start work on it.
- Got over my K8s issue, helped by Jeena.
- Gave a PGP basics tutorial, helped dvpita to generate key.
- What I plan to do this week
- Seek feedback on v3 of CI arch doc, after minor updates.
- Set up Argo in a K8s cluster somewhere.
- Update engprod list on PGP keysigning at offsite.
- Book club.
- Talk with people going to TechConf about new CI.
- What I'm blocked on
- Other?
- not being well this week
- away Thursday evening Finnish time for Rust meetup
Mukunda[edit]
- What I planned vs did last week
Done Branch REL1_34 with Dan
Done Work on branch.py (monthly achievements!)
Not done Phabricator Deploy
Not done Work on getting phabricator to notify you when attempting to submit a duplicate phatality report.
- What I plan to do this week
- Phatality Deployment
- Phabricator Deployment
- Work on Phabricator migrations and disaster recovery stuff
- Come up with some topics for the focused patch triage (action item for code review working group)
- What I'm blocked on
- Other?
Tyler[edit]
- What I planned vs did last week
Done Leveling
Done OKRs
- What I plan to do this week
- connect with SRE re New CI
- Set date for Gerrit move to new server
- OKR...fiddling
- What I'm blocked on
- Other?
Zeljko[edit]
- What I planned vs did last week
Done T213268 Upgrade webdriverio to version 5 in mediawiki/core - by Timo, I was a reviewer
In progress T234002 Make MediaWiki Wdio tests less slow (Sept 2019)
In progress T199113 All repositories with Selenium tests should use wdio-mediawiki
Done T234311 Fix and restore daily browser tests for TwoColConflict
Not done T232638 Add accessibility check for Homepage
- What I plan to do this week
- Stare at SafeGuard Letter. For hours. Then, sign it.
- T199113 All repositories with Selenium tests should use wdio-mediawiki
- T234610 ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73
- T232638 Add accessibility check for Homepage
- T234085 WMTC19: Themes/Focus Areas Proposals
- T234088 Theme: Testing
- T234635 Wikimedia Technical Conference 2019 Session: System level testing: patterns and anti-patterns with Selenium
- T234088 Theme: Testing
- What I'm blocked on
- I can't reproduce CI failure on my machine, making it hard to fix :/ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TemplateWizard/+/540894#message-fa48dda448cd609a6ca513c807ee3e47aaa9b827
- Other?
- Not blocked but stuck: T233960 Error: error installing: the server could not find the requested resource
- At a local conference on Friday
- Running a half maratnon on Sunday. I've run several, but the last one was 12 years ago.