Wikimedia Release Engineering Team/Trainsperiment week

From mediawiki.org

πŸš‚πŸ§ͺ Trainsperiment Week - March 21, 2022.

Four deployment trains. One week. Six nervous Release Engineers!

What is this!?[edit]

The "deployment train" is the name of the process where the Release Engineering Team (RelEng) deploys between 150–950 patches to the ~980 Wikimedia wikis, in 3 stages. There is usually one train each week.

There will be four deployment trains during the week of Monday, 21 March 2022.

This is a one-week experiment to gather data for possible future changes.

If you have ideas or thoughts, please add them on the talk page. πŸ’‘

Visualization[edit]

These images show the difference between the regular process and this new experimental process:

πŸ€” Unknowns[edit]

Here is a list of things we are thinking about. If you can help with any of these: please talk to us!

  • How will more trains affect caches and site performance? (T303758)
  • Are developers going to need to be more mindful of forward and backward compatibility? (T303759)
  • How will train-rollback work with blockers? This one is on us, but we’ll be figuring it out as we go. (T303747)
  • Will blocker tasks be too confusing with so many trains?
  • Is this sustainable?

πŸ† Goals[edit]

Our goal is a shorter time between when developers write code and when that code gets live traffic. We believe the benefits of that are:

  • A more accurate mental model of the current state of the code that is "live" will make development easier
  • A smaller amount of code per release, will make the releases easier to debug and test
  • It will make it easier for developers to fix bugs in a release if the context is still fresh in their heads

πŸ“… Schedule[edit]

Monday, 21 March 2022 #

Window Date Time Event Signup to join the hangout
Branch Cut 21 Mar 2022 02:00UTC 1.39.0-wmf.1 Branch Cut N/A (it's automatic)
UTC AM Train 21 Mar 2022 09:00UTC
  • (+0 minutes) 1.39.0-wmf.1 Group0
  • (+30 minutes) 1.39.0-wmf.1 Group1
  • tcipriani@wikimedia.org
  • ...
Branch Cut 21 Mar 2022 12:00UTC 1.39.0-wmf.2 Branch Cut N/A (it's automatic)
UTC PM Train 21 Mar 2022 18:00UTC
  • (+0 minutes) 1.39.0-wmf.1 Group2

  • (+30 minutes) 1.39.0-wmf.2 Group0 (Ξ” 10 hours)
  • (+45 minutes) 1.39.0-wmf.2 Group1
  • (+90 minutes) 1.39.0-wmf.2 Group2
  • tcipriani@wikimedia.org
  • ...

Tuesday, 22 March 2022 #

Window Date Time Event Signup to join the hangout
Branch Cut 22 Mar 2022 02:00UTC 1.39.0-wmf.3 Branch Cut N/A (it's automatic)
UTC AM Train 22 Mar 2022 09:00UTC
  • (+0 minutes) 1.39.0-wmf.3 Group0 (Ξ” 14 hours)
  • (+15 minutes) 1.39.0-wmf.3 Group1
  • (+60 minutes) 1.39.0-wmf.3 Group2
  • tcipriani@wikimedia.org
  • ...

Wednesday, 23 March 2022 #

Window Date Time Event Signup to join the hangout
Branch cut 23 Mar 2022 02:00UTC 1.39.0-wmf.4 Branch cut N/A (it's automatic)
UTC PM Train 23 Mar 2022 18:00UTC
  • (+0 minutes) 1.39.0-wmf.4 Group0 (Ξ” 20 hours)
  • (+15 minutes) 1.39.0-wmf.4 Group1
  • (+60 minutes) 1.39.0-wmf.4 Group2
  • tcipriani@wikimedia.org
  • ...

Thursday, 24 March 2022 #

Window Date Time Event Signup to join the hangout
UTC AM Train 24 Mar 2022 08:00UTC
  • Cleanup and catchup
  • tcipriani@wikimedia.org
  • ...
UTC PM Train 24 Mar 2022 18:00UTC
  • Cleanup and catchup
  • tcipriani@wikimedia.org
  • ...

Outcomes[edit]

A postmortem on the experiment including summaries of feedback received has been published as What We Learned from Trainsperiment Week in the Release Engineering team's blog Doing the needful.