Wikimedia Release Engineering Team/Trainsperiment week

From mediawiki.org

πŸš‚πŸ§ͺ Trainsperiment Week - March 21, 2022.

Four deployment trains. One week. Six nervous Release Engineers!

What is this!?

The "deployment train" is the name of the process where the Release Engineering Team (RelEng) deploys between 150–950 patches to the ~980 Wikimedia wikis, in 3 stages. There is usually one train each week.

There will be four deployment trains during the week of Monday, 21 March 2022.

This is a one-week experiment to gather data for possible future changes.

If you have ideas or thoughts, please add them on the talk page. πŸ’‘

Visualization

These images show the difference between the regular process and this new experimental process:

πŸ€” Unknowns

Here is a list of things we are thinking about. If you can help with any of these: please talk to us!

  • How will more trains affect caches and site performance? (T303758)
  • Are developers going to need to be more mindful of forward and backward compatibility? (T303759)
  • How will train-rollback work with blockers? This one is on us, but we’ll be figuring it out as we go. (T303747)
  • Will blocker tasks be too confusing with so many trains?
  • Is this sustainable?

πŸ† Goals

Our goal is a shorter time between when developers write code and when that code gets live traffic. We believe the benefits of that are:

  • A more accurate mental model of the current state of the code that is "live" will make development easier
  • A smaller amount of code per release, will make the releases easier to debug and test
  • It will make it easier for developers to fix bugs in a release if the context is still fresh in their heads

πŸ“… Schedule

Monday, 21 March 2022 #

Window Date Time Event Signup to join the hangout
Branch Cut 21 Mar 2022 02:00UTC 1.39.0-wmf.1 Branch Cut N/A (it's automatic)
UTC AM Train 21 Mar 2022 09:00UTC
  • (+0 minutes) 1.39.0-wmf.1 Group0
  • (+30 minutes) 1.39.0-wmf.1 Group1
  • tcipriani@wikimedia.org
  • ...
Branch Cut 21 Mar 2022 12:00UTC 1.39.0-wmf.2 Branch Cut N/A (it's automatic)
UTC PM Train 21 Mar 2022 18:00UTC
  • (+0 minutes) 1.39.0-wmf.1 Group2

  • (+30 minutes) 1.39.0-wmf.2 Group0 (Ξ” 10 hours)
  • (+45 minutes) 1.39.0-wmf.2 Group1
  • (+90 minutes) 1.39.0-wmf.2 Group2
  • tcipriani@wikimedia.org
  • ...

Tuesday, 22 March 2022 #

Window Date Time Event Signup to join the hangout
Branch Cut 22 Mar 2022 02:00UTC 1.39.0-wmf.3 Branch Cut N/A (it's automatic)
UTC AM Train 22 Mar 2022 09:00UTC
  • (+0 minutes) 1.39.0-wmf.3 Group0 (Ξ” 14 hours)
  • (+15 minutes) 1.39.0-wmf.3 Group1
  • (+60 minutes) 1.39.0-wmf.3 Group2
  • tcipriani@wikimedia.org
  • ...

Wednesday, 23 March 2022 #

Window Date Time Event Signup to join the hangout
Branch cut 23 Mar 2022 02:00UTC 1.39.0-wmf.4 Branch cut N/A (it's automatic)
UTC PM Train 23 Mar 2022 18:00UTC
  • (+0 minutes) 1.39.0-wmf.4 Group0 (Ξ” 20 hours)
  • (+15 minutes) 1.39.0-wmf.4 Group1
  • (+60 minutes) 1.39.0-wmf.4 Group2
  • tcipriani@wikimedia.org
  • ...

Thursday, 24 March 2022 #

Window Date Time Event Signup to join the hangout
UTC AM Train 24 Mar 2022 08:00UTC
  • Cleanup and catchup
  • tcipriani@wikimedia.org
  • ...
UTC PM Train 24 Mar 2022 18:00UTC
  • Cleanup and catchup
  • tcipriani@wikimedia.org
  • ...

Outcomes

A postmortem on the experiment including summaries of feedback received has been published as What We Learned from Trainsperiment Week in the Release Engineering team's blog Doing the needful.