Wikimedia Release Engineering Team/Checkin archive/2026-07-08

2026-07-08

• Last time

📋 Agenda

• Wins
• Vacations/Important dates
• Clinic duty and Train update
• Team Discussion:

🏆 Wins/winterrogation

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments

Jul 2026

• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1209 config.py: Fix default mw_web_clusters
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1210 main.py: Only announce rollback when k8s deployment is enabled
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305483 scap.cfg.erb: Add jobrunner to beta mw_web_clusters list
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302910 modules/beta/files/wmf-beta-update-databases.py: Keep update.php jobs topped up
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302978 modules/profile/files/puppet/bin: cleanup puppet SSL on CA server change
• https://github.com/yaoyuannnn/gerritlab/pull/87 2.1.0: Ensure remote branches are ready before creating MRs
• https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/183 scap.cfg: mediawiki_image_extra_packages netcat -> netcat-openbsd
• https://gerrit.wikimedia.org/r/c/mediawiki/core/+/827589 rebuildLocalisationCache.php: Print the reason for rebuilding a language [Created Aug 2022]

• Fixed https://phabricator.wikimedia.org/T429662 No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep
• Fixed https://phabricator.wikimedia.org/T430384 Last Puppet run was over 24 hours ago on instance deployment-echostore02 in project deployment-prep
  • Both of these were due to an ever-growing container log filling up the filesystem. Need to find a permanent solution

• Rolled fix for https://phabricator.wikimedia.org/T425165 Cannot select REL1_46 on PatchDemo (thanks for helping dig for repo paths, Jeena)

• Documented using the Sortinghat GraphQL API with curl examples on https://www.mediawiki.org/w/index.php?title=Community_metrics&diff=8460816&oldid=8445731

---

WE ARE HERE (2026-07-08)

Scap

• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1212 backport: Remove extension.json from L10N_REBUILD_PATH_PATTERNS
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1214 requirements: Bump vulnerable dependencies to address Dependabot alerts
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1215 package-lock.json: Updates from npm audit fix
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1217 requirements: Bump fastapi and starlette to fix CVE-2025-62727

Buildkit

• https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/81 Dockerfile: Bump base bookworm image tag to 20260705
• https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/619 staging/staging.tfvars: Use buildkitd wmf-v0.31.1
• https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/620 production/prod.tfvars: Use buildkitd wmf-v0.31.1

GerritLab

• https://github.com/yaoyuannnn/gerritlab/pull/88 2.1.1: Retry MR creation while source branch is pending
• https://github.com/yaoyuannnn/gerritlab/pull/89 Allow overriding the GitLab API host via git config

Misc

• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306745 profile::puppet::agent: write pinned CA cert with a trailing newline

• Merged Tyler's thing for de-confusifying the "open wiki" button on patchdemo
• Added RSS extension to patchdemo (at least I think I got everything)
• Moved some more patchdemo stuff into Slim framework
• Routine Phab deploy: https://phabricator.wikimedia.org/T431440

• Jenkins to new host: https://phabricator.wikimedia.org/T418521
• Zuul ran a build within a k8s pod and uploaded artifacts ( https://zuul.wikimedia.org/t/wikimedia/builds )
• Zuul web app fixed, its API calls were erroneously cached by our CDN

• Wrote a patch to update Conduit calls in CodeReviewBot, CI & tests is awesome, didn't have to set up stuff
• Andre poked more validation stuff in upstream Phorge, no win, just more issues

📅 Vacations/Important dates

https://office.wikimedia.org/wiki/Staff_handbook/Benefits#US_Holidays https://office.wikimedia.org/wiki/People_Corner/Global_Guidelines#Holidays

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off

This week and next

• This week: Jaime
• Next week: Antoine, Jaime, Andre

• Jul 3: U.S. Fourth of July Holiday (observed) (US Staff with reqs)
• Jul 6 Mon: Andre public holiday
• Jul 6–Aug 31: Tyler sabbatical!
• Jul 9: Brennen out afternoon
• Jul 1-14: Jaime Vacation
• July 13th-17th: Antoine extended holiday (Bastille's day)
• Jul 15-31: Jaime Spain (internet less stable)
• Jul 14Tue - 21Tue: Andre OoO (Spain)
• July 24 - Aug 5ish: Jeena judo training in Japan
• July 21th-28th - Antoine in Paris, working there + meeting Wikimania folks in the evening
• Aug 3-7: Dan vacation
• Aug 10: International Day of the World’s Indigenous Peoples (observed) (Global holiday)
• Aug 11 Tue: Andre OoO
• Aug 21 Fri: Andre maaaaybe OoO
• Aug 30-Sep 9: Brennen in BRC
• September - October: Bryan sabattical
• Sep 07: U.S. Labor Day (US Staff with reqs)
• Sep 16-20: Brennen 🪕
• Oct 12: Columbus Day (also Indigenous Peoples' Day) (US Staff with reqs)
• Nov 11: Veterans' Day (US Staff with reqs)
• Nov 26: U.S. Thanksgiving (US Staff with reqs)
• Nov 27: Day after U.S. Thanksgiving (US Staff with reqs)
• Dec 24–Jan 1 2026: End of year holiday (Global holiday)

🔥🚂 Train & Clinic duty

https://versions.toolforge.org/

https://train-blockers.toolforge.org/

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Clinic_duty

PAST

• 27 Apr – 1.46.0-wmf.26 – Jeena, Dan
  • Out: Brennen, Andre
• 04 May – 1.47.0-wmf.1 – Brennen, Jeena https://phabricator.wikimedia.org/T423910
  • Out:
• 11 May – 1.47.0-wmf.2 – Andre, Brennen https://phabricator.wikimedia.org/T423911
  • Out: Antoine
  • Learnings: EventBus errors become UBNs
• 18 May – 1.47.0-wmf.3 – Antoine, Andre https://phabricator.wikimedia.org/T423912
  • Out:
• 25 May – 1.47.0-wmf.4 – Jaime, Antoine https://phabricator.wikimedia.org/T423913
  • Out:
• 01 Jun – 1.47.0-wmf.5 – Ahmon, Jaime https://phabricator.wikimedia.org/T423914
  • Out:
• 08 Jun – 1.47.0-wmf.6 – Dan, Jaime https://phabricator.wikimedia.org/T423915
  • Out: Ahmon
• 15 Jun – 1.47.0-wmf.7
  • Train: Jeena
  • Clinic Duty: Dan
  • Out: Andre (Thu, Fri), US staff w/reqs (Fri)

• 22 Jun - 1.47.0-wmf.8
  • Train: Brennen
  • Clinic Duty: Jeena
  • Out: Ahmon (Mon)

• 29 Jun - 1.47.0-wmf.9
  • Train: Andre
  • Clinic Duty: Brennen
  • Out: Dan, Jaime

NOW and NEXT

• 06 Jul – 1.47.0-wmf.10
  • Train: Antoine
  • Clinic Duty: Andre
  • Out: Jaime, Andre (Mon)

• 13 Jul – 1.47.0-wmf.11
  • Train: Jeena
  • Clinic Duty: Brennen (covering for Antoine)
  • Out: Jaime; Andre (Wed-Fri)

• 20 Jul – 1.47.0-wmf.12
  • Train: Ahmon
  • Clinic Duty: Jeena
  • Out: jeena; Andre (Mon-Tue)

• 27 Jul – 1.47.0-wmf.13
  • Train: Dan
  • Clinic Duty: Ahmon
  • Out: jeena

• 03 Aug – 1.47.0-wmf.14
  • Train: Jaime
  • Clinic duty: Jeena
  • Out: Dan

• 10 Aug – 1.47.0-wmf.15
  • Train: Brennen
  • Clinic duty: Jaime
  • Out: Everyone Monday, Andre Tue

• 17 Aug – 1.47.0-wmf.16
  • Train: Andre
  • Clinic duty: Brennen
  • Out:

• 24 Aug – 1.47.0-wmf.17
  • Train: Antoine
  • Clinic duty: Andre
  • Out:

• 31 Aug – 1.47.0-wmf.18
  • Train: Ahmon
  • Clinic duty: Antoine
  • Out: Brennen

• 07 Sep – 1.47.0-wmf.19
  • Train: Dan
  • Clinic duty: Ahmon
  • Out:

• 14 Sep – 1.47.0-wmf.20
  • Train: Jaime
  • Clinic duty: Dan
  • Out:

Team Discussions

• Did Tyler get an extension on ITC? Got an ITC email saying "08/07" (no clue how to parse that)
  • Antoine: Tyler told me mine would happen in September, so I guess he arranged to have all ITCs pushed back
• Andre says: http://phabricator.wikimedia.org/T240266#11909900 kinda welcomes an answer from RelEng
  • Looks like Antoine responded here

Sabbatical plans

• Doc: https://docs.google.com/document/d/1jGrRMw7CzOLz7DzeNYLVmXTo8Z-vHi3XPaIZjbourj4/edit?tab=t.0
• Contact: https://office.wikimedia.org/wiki/User:TCipriani_(WMF)/Continuity (still not aligned with ^, FYI)

• Zuul merger is super backed up
  • after a certain amount of time and as a last resort, we can restart Zuul to flush the Gearman queues
  • TODO: restart Zuul (8 hrs backed up atm) (trying the readonly for merger function queue first)
  • TODO: add documentation to our runbooks about the Zuul merger issue and what to do
  • https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions

• Zuul v14 update
  • some outstanding warts
    • Issues with namespace based labels (they weren't completely implemented in new Zuul)
    • Uploading of artifacts to swift wasn't working