Wikimedia Release Engineering Team/Checkin archive/20191002
Appearance
2019-10-02
[edit]Vacations/Important dates
[edit]- September 23–October 4: JR Vacation
- October 8: Željko holiday (Independence Day)
- October 11: Željko conference (https://2019.webcampzg.org/ )
- October 14 (Indigenous Peoples' Day): US Staff
- November 11 (Veterans' Day): US Staff
- November 12–15: TechConf19
- November 16–20: Offsite
- November 28–29 (Thanksgiving): US Staff
- December 6: Lars (Finnish Independence Day)
- December 25–31 (Christmas): US Staff
- December 25–26: Lars, Christmas
- 1 January (New Year's Day): US Staff, Lars
- 20 January (Martin Luther King Jr. Day): US Staff
Rotating positions
[edit]Train
[edit]- Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R
- June 24 - wmf.11 - Jeena (with Mukunda)
- July 1 - wmf.12 - No train (Fourth of July)
- July 8 - wmf.13 - Jeena
- July 15 - wmf.14 - Lars (with Antoine)
- July 22 - wmf.15 - Lars
- July 29 - wmf.16 - Brennen (with Tyler)
- Aug 5 - wmf.17 - Brennen
- Aug 12 - wmf.18 - No Train (Wikimania)
- Aug 19 - wmf.19 - Zeljko 😱
- Aug 26 - wmf.20 - Zeljko 😭
- Sep 2 - wmf.21 - Antoine
- Sep 9 - wmf.22 - Antoine
- Sep 16 - wmf.23 - Mukunda
- Sep 23 - wmf.24 - Mukunda
- Sep 30 - wmf.25 - Tyler Dan
- 7 Oct - wmf.26 - Tyler Dan
- 17 Oct - 1.35.0-wmf.1 - TBD
- 21 Oct - wmf.2 - TBD
- 28 Oct - wmf.3 - TBD
- 4 Nov - wmf.4 - TBD
- 11 Nov - wmf.5 - TBD (? No train – TechConf ?)
- Who is available from EngProd + CPT + SRE?
- 18 Nov - No train – EngProd offsite
- 25 Nov - wmf.7 - Antoine? (? No train – Thanksgiving ?)
- EU Train week?
- Or just run with a delayed "Thursday" to the next Monday?
SoS
[edit]- Aug 14 onward: Zeljko
- SoS on 2019-10-09 is at the same time as Tech All Hands
Team Business
[edit]Timespent spreadsheet
[edit]- For the avoidance of doubt: fill out the sheet week number for the previous week
- spreadsheet for week 39 (9/23): https://docs.google.com/spreadsheets/d/1UEmjIbv6voTF5bOnHdjOltw47AwPDtnmUo0h78wgYDY/edit?pli=1#gid=131825410
November Offsite/Summit
[edit]- TechConf19: November 12–15
- Offsite: November 16–20
- Arrive the 15th
- 16th is our fun day/outing
- 17–20 are working days
- 21st fly home
- https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
Book club
[edit]- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
- 2019-10-15
- Read the blog post: http://blog.codepipes.com/testing/software-testing-antipatterns.html
- Let's discuss! https://etherpad.wikimedia.org/p/releng-book-club
- 2019-10-15
Monthly reflection on accomplishments - September '19 edition
[edit]- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
- Add as you have them!
- Scap 3.12.1-1 released/deployed
- Refactored Zuul layout to use per-branch pipelines
quibble -c
Lets you run arbitrary code against a working MediaWiki install- The phabricator "Report Error Code" form (https://phabricator.wikimedia.org/maniphest/task/edit/form/46/ ) has been updated with separate fields for the stack trace and error code/request id.
- T232608 Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days
- Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ (first step towards a saner config)
- MediaWiki PHP support target modernised from 7.0+ to 7.2+ for 1.34 onwards. https://phabricator.wikimedia.org/T228342
- Quibble 0.0.35 release
- 1.34.0-wmf.24 branch cut was done /mostly/ with branch.py instead of make-wmf-branch.php (some small bugs remain to work out but it's very close)
- Creating accounts was broken on beta cluster since 2019-09-08. It was fixed today (2019-09-25). https://phabricator.wikimedia.org/T232796
- Phatality extension for Kibana deployed to production and used for reporting production errors into Phabricator.
- Train blocker tasks created for 1.35.0-wmf.1-1.35.0-wmf.25
- MediaWiki dev images are now automatically created as part of postmerge via the pipeline
Incoming/Needs attention
[edit]- Pipeline: https://phabricator.wikimedia.org/T233316#5536374
- CPT secrets: https://phabricator.wikimedia.org/T233092
New Month, New Workboard!
[edit]- Please move from 201909 -> 201910 (and move up to TODO/close invalidate/whatever liberally)
- 201909 https://phabricator.wikimedia.org/project/view/4253/
- 201910 https://phabricator.wikimedia.org/project/view/4311/
- Let me know in 1:1s if you're stuck/have a question about how to treat a task
Proposed New Rotating Position
[edit]- Clinic Duty!
- Analogous to:
- Proposed scope
- Triage tasks on #together INBOX/Release-Engineering-Team-TODO INBOX/Release-Engineering-Team-TODO (201XXX) INBOX
- Handle "hey releng"-type requests in #wikimedia-releng (@meatbags for brennen)
- +2 common requests (add x@y.com to ci whitelist)
Scrum of Scrums for next week
[edit]Internal team notes, not for pasting into the SoS notes
[edit]Incoming from last week (aka this morning)
[edit]- The Growth team is planning deployments for the newcomer tasks projects and we need to know November black out dates
- CPT: Create a way to share a secret between MediaWiki and the testing framework in CI phab:T233092
- Kartotherian folks: phab:T233316
- Not blocked, but nice-to-have for Fundraising tech: php7.0 docker image for CRM tests phab:T230446
- Fundraising emails to English-speaking countries started going out this week - that traffic goes through donate.wikimedia.org (on the main cluster) before it hits payments-wiki (on the fr cluster), so let's nobody break it.
Outgoing next week
[edit]- Blocked:
- Blocking:
- Updates:
- Train Health
- Last week: 1.34.0-wmf.25 - phab:T220750
- This week: 1.35.0-wmf.1 - phab:T233849
- Next week: 1.35.0-wmf.2 - phab:T233850
- Train Health
Callouts
[edit]Train status and happenings
[edit]OKRs, Goals, Betterworks, Oh my!
[edit]- Concerns? Questions no one can answer?
- Additionally (yes, that's right)...Team Goals
Standup!
[edit]- Make sure your tasks are correct on our kanban board!
Antoine
[edit]- What I planned vs did last week
- Had Castor instance relocated to have more CPU https://phabricator.wikimedia.org/T232646 \o/
- Extract Selenium to standalone job https://phabricator.wikimedia.org/T232759
- Gerrit and java GC debugging (some bot was still hammering the master) https://phabricator.wikimedia.org/T234328
- Found a slowness bug in php-xdebug from component/php72 ( https://phabricator.wikimedia.org/T234418 )
- What I plan to do this week
- More extraction of Selenium to standalone jobs ( https://phabricator.wikimedia.org/T232759 )
- Teach Quibble to record duration of each stages and report at end of builds (partly done)
- OKR?
- Reviews for Quibble
- Stretch VM are slower than the Jessie ones https://phabricator.wikimedia.org/T232706
- What I'm blocked on
- Status of HHVM in production? Which implies: can we drop HHVM from CI?
- Other?
- Some cloudvirt are unreasonably slow. Still going on
- https://phabricator.wikimedia.org/T225713 "CPU scaling governor fix up"
- https://phabricator.wikimedia.org/T223971 "old cloudvirt are CPU slow"
- Logstash on beta has been broken for months https://phabricator.wikimedia.org/T233134
- Some cloudvirt are unreasonably slow. Still going on
Brennen
[edit]- What I planned vs did last week
Done generate MediaWiki + Apache dev image from pipeline (!!!)
Not done Send Lars my PGP key, after I figure out what's wrong with my PGP key
Done Read Lars's latest CI document
Done Final(?) CI WG meeting on Friday
Done Investigate whether https://phabricator.wikimedia.org/T229280 is really still happening
In progress Focus on dev environment questions
- Unplanned: Reviewed some pipelinelib patches
- What I plan to do this week
- Dev environment experimentation
- Pair with Jeena on generating some service dev images in pipeline
- T234391 Point deployment-charts/mediawiki-dev at latest dev image published by pipeline
- Put OKRs in Betterworks
- Read for book club
- Figure out what violence Enigmail and Keybase did to my GPG config
- What I'm blocked on
- Other?
Dan
[edit]- What I planned vs did last week
In progress Seakeeper K8s Proposal
- [yay!]++
- What I plan to do this week
- What I'm blocked on
- Other?
Greg
[edit]- What I planned vs did last week
Done Tech-mgt F2F, meet Grant, new CTO
Done Leveling...
- What I plan to do this week
- TechConf session drafting
- OKR drafting, aligning
- Goal setting cross EngProd
- What I'm blocked on
- The announcement for EngProd creation :) (just waiting, not actually blocking anything other than nice feelies)
- Other?
James
[edit]- What I planned vs did last week
Partially done Vacation all week – I worked on Thursday anyway, because I was bored.
- What I plan to do this week
- Supporting 1.34 release branch cut/etc.
- OKR fun.
- HHVM removal from CI and later MW
- Static configuration progress https://phabricator.wikimedia.org/T223602
- Iterating from CodeReview metrics/KPI proposal to make some for CI
- What I'm blocked on
- Seeking input from deployers/configurators on plans for static configuration changes
- Other?
- —
Jean-Rene
[edit]- What I planned vs did last week
- vacation
- What I plan to do this week
- vacation
- What I'm blocked on
- Other?
Jeena
[edit]- What I planned vs did last week
Done Update parsoid blubberfile to new version
Done local-charts meeting
Not done Get parsoid image built in CI. Same for restbase?
Done Fix some issues with mediawiki chart
Done Fix some issues with parsoid chart
In progress bring up issue with deployment-charts about templates with same name
Done Add ideas to local dev brainstorm
Done local dev meeting
Done docker sig meeting?
- What I plan to do this week
- make example of deployment charts template problem for Alexandros
- OKR writing
- local dev meeting & prep
- Work with Brennen to get parsoid image built in CI. Same for restbase? Good question.
- Explore what's needed for new php parsoid service in terms of local-charts/deployment-charts
- Reading for book club
- What I'm blocked on
- Other?
Lars
[edit]- What I planned vs did last week
Not done Get feedback on v3 of CI arch doc. Process it.
Done Read up on Zuul dependent pipelines.
Done With CI WG, decide on CI engine for future CI. Make plan on what next.
Not done Update engprod list on PGP keysigning at offsite.
Not done Read the book club article. Let subconsicous start work on it.
Done filled in my OKRs in betterworks
- What I plan to do this week
- Seek feedback on v3 of CI arch doc.
- Set up Argo in a K8s cluster somewhere.
- Update engprod list on PGP keysigning at offsite.
- Read the book club article. Let subconsicous start work on it.
- What I'm blocked on
- K8s - could do with some help
- Other?
Mukunda
[edit]- What I planned vs did last week
Done Train
Done Get familiar with betterworks
Done Deploy phatality to production
Done Develop some OKRs
- What I plan to do this week
- Branch REL1_34 with Dan
- Work on branch.py
- Phabricator Deploy
- Work on getting phabricator to notify you when attempting to submit a duplicate phatality report.
- What I'm blocked on
- Other?
Tyler
[edit]- What I planned vs did last week
Done tech-mgr f2f
- What I plan to do this week
- Goals
- Meetings
- Gerrit1001
- What I'm blocked on
- Other?
Zeljko
[edit]- What I planned vs did last week
Done T232598 Update existing Selenium documentation
Not done T232638 Add accessibility check for Homepage (GrowthExperiments)
- What I plan to do this week
- T213268 Upgrade webdriverio to version 5 in mediawiki/core
- T234002 Make MediaWiki Wdio tests less slow (Sept 2019)
- T199113 All repositories with Selenium tests should use wdio-mediawiki
- T234311 Fix and restore daily browser tests for TwoColConflict
- T232638 Add accessibility check for Homepage
- What I'm blocked on
- T233960 Error: error installing: the server could not find the requested resource
- SoS on 2019-10-09 is at the same time as Tech All Hands
- Other?
- Should I be working on individual goals? apparently not
- October 1 was 7 (seven!) years I've been working for WMF