Wikimedia Release Engineering Team/Pretrain
|
Pretrain
Automated deployment of MediaWiki code and config into a production Kubernetes cluster
|
Pretrain is an environment supporting manual and automated integration and regression testing of changes before those same changes reach the Wikimedia movement's content wikis like Wikidata and English Wikipedia via the Deployment train process.
The project has its roots in discussions about T215217: deployment-prep (beta cluster): Code stewardship request and the needs of the Wikimedia Quality Services & Wikimedia Test Platform teams.
 | This page in a nutshell: By the end of September 2026 [1], when you visit test.wikipedia.org you will be using code that merged last night. |
What is changing?
[edit]- Every night we bundle everything that has merged for MediaWiki core, 193 extensions, and 8 skins into a new release branch named
wmf/next. - Monday through Thursday we will deploy
wmf/nextto Pretrain wikis. - test.wikipedia.org (
testwiki) will be moved from Group0 (new code on Tuesdays) to Pretrain (new code Monday-Thursday). - Integration and regression testing workflows for the Wikimedia Quality Services team will begin moving from Beta Cluster to Pretrain.
- All users of Beta Cluster are encouraged to examine their workflows to determine if Pretrain is a better fit for the work being done.
Timeline
[edit]- July-October 2024:
Done Created a process and automation to manage the wmf/nextrelease branch. (final report) - November 2024-April 2025: Done Consulted with multiple groups and individuals to establish the strict requirements for production deployment. (final report)
- July-September 2025: Done Updating past work to align with production requirements and additional preparation for a future production deployment. (final report)
- January-March 2026: Done Document reliability risks and compensating controls for those risks to establish consensus on key decisions needed for deployment.
- April-September 2026:
In progress Deploy wmf/nextto a Kubernetes cluster via automated Pretrain deployments and during manual backport deployments.
What should I do?
[edit]- Users of testwiki
- Be aware that code is going to change more often than it does today. Code will change nightly during the US work week.
- Developers of anything that rides the Deployment Train
- You can use testwiki as a testing environment for code that merged yesterday.
- Developers of user scripts and gadgets
- You can use testwiki as a testing environment against code that merged yesterday.
- Developers of templates and Lua modules
- You can use testwiki as a testing environment against code that merged yesterday.
FAQ
[edit]When will my code reach Pretrain wikis?
[edit]Code is deployed on Pretain wikis nightly during the US work week.
| Sunday | Monday | Tuesday | Wednesday | Thursday | Friday | Saturday | |
|---|---|---|---|---|---|---|---|
| Train |
|
|
|
||||
| Pretrain |
|
|
|
|
Merging any day other than Monday will give you extra time to test your code on Pretrain before it rolls out to other groups:
- Monday: 0 days on Pretrain
- Tuesday: 6 days on Pretrain
- Wednesday: 5 days on Pretrain
- ...
Examples:
- Code merged on Monday
- Pretrain early Tuesday UTC morning, Group0 wikis Tuesday UTC evening, mediawiki.org on Wednesday, en.wikipedia.org on Thursday
- Code merged on Wednesday
- Pretrain early Thursday UTC morning, Group0 wikis next Tuesday, mediawiki.org next Wednesday, en.wikipedia.org next Thursday
- 5 days on Pretrain before reaching mediawiki.org
Why only a daily deployment?
[edit]We are learning to walk before we run. We would like to deploy more frequently and to more wikis in the future.
Is anything changing about backports?
[edit]No
Is anything changing about security patches?
[edit]No
Is anything changing about who does deployments?
[edit]No
What are the risks associated with faster deployments?
[edit]In interviewing developers, they’ve raised concerns about atomicity, feature flagging, and catastrophic bugs.
As we’ve dug into each concern, all the problems we currently have we will still have, but we have yet to find new problems. That is, we are merely doing what we do every Tuesday every day.
The biggest risk is lack of awareness of this change.
Can I do automated testing on Pretain wikis?
[edit]Probably, but be mindful of things that create noise outside of the test wikis like SUL account changes, edits that look like vandalism, and the stated purpose of the wiki you are testing against.
Why are we making these changes?
[edit]We have known for many years that validation testing in Beta Cluster is complicated by the stability of the services hosted there. When bugs are found an additional investigation is often needed to determine if the problem is caused by newly introduced code, misconfiguration of the platform, or failures unrelated to the system under test.
Conversations began in late 2023 about what it might take to create a more stable "production-like" environment for the test engineers to use. It is undoubtedly true that the most "production-like" environment is the production network itself. Thinking from this point of view led to asking questions about what barriers prevent us from deploying software more frequently to a controlled set of wikis where automated and manual tests could be run. A related question is if this environment could be stable enough to host a set of content wikis like office.wikimedia.org, wikitech.wikimedia.org, and www.mediawiki.org where the on-wiki communities could also become part of finding bugs closer to the time they are introduced by developers.
By testing and using MediaWiki more frequently before a particular version reaches the Wikipedia wikis we hope to "shift left" and find defects closer to the time where developers introduced them. Ultimately we hope the fewer bugs make it all the way to the large content wikis before they are found.
Project goals
[edit]Workflow goals
[edit]- Enable testing of pre-train ("next") branches of MediaWiki, skins, and extensions in a stable environment where newly discovered defects are more likely to be the result of the next branches than problems with support services or configuration in the environment itself.
Technical goals
[edit]- Done Automate MediaWiki OCI image creation based on a timer or similar trigger.
- In progress Create an environment in the production network for running pre-train MediaWiki versions.
- In progress Automate MediaWiki OCI container deployment into the pre-train environment.
- Done Enable overriding any staged wikiversions.json when
scapis determining which MediaWiki versions need to be included in a container (allow, but do not require single version builds). - Done Enable overriding an in-container wikiversions.json with a hard coded MediaWiki version inside of the container.
- Done Enable creation of MediaWiki containers from arbitrary staging directories so that a single deployment or CI server can be used to build as many variant containers as we find need for.
Out of scope
[edit]Workflow out of scope
[edit]- Enabling testing of pre-release services and configuration not managed by
scapis out of scope.
Technical out of scope
[edit]- Replacing the weekly train progression with continuous delivery to all wikis is out of scope.
- Building an image for every commit to a Train deployed repo/submodule is out of scope.
- Keeping deployment-prep working in the face of production's migration to Kubernetes and containers as the MediaWiki deployment and runtime solution is out of scope.
- Building a chain of images starting from public files only to produce images that can be used outside of production is out of scope.
- Reimagining how configuration is delivered into MediaWiki containers is out of scope.
- Runtime support for single version images beyond minimum functionality needed to support building and operating Pretrain directly is out of scope.
Design agreements
[edit]The Pretrain environment will be created by adding new releases to the staging Kubernetes cluster. These releases will either be multiple role-specific services under a new mw-pretrain namespace or additional "pretrain" services in existing role related namespaces such as mw-{cron,jobrunner,mwscript,web}. A "release" in this parlance is an instance of a Helm chart running in a Kubernetes cluster. These releases will generally be known as pretrain in documentation including this proposal.
Pretrain will use OCI container images containing a single wmf/next MediaWiki version. A new wmf/next image will be created once per day starting at 01:00 UTC. This image will be deployed via an automated process triggered by a scheduled task running from the active deployment server Monday through Thursday at 02:00 UTC. Additional workflows for rollback and ad hoc deployments will be available to authorized parties via scap and SpiderPig as appropriate.
Deployment of both MediaWiki configuration (operations/mediawiki-config.git) and security patches done via scap from a deployment server will also trigger new pretrain deployments. These deployments will not happen in-process with the other wiki updates. Instead they will trigger an asynchronous build and deployment workflow. We may choose to de-duplicate triggering events so that we only deploy pretrain once as the result of a typical backport or security window.
The initial pretrain wiki will be testwiki. We expect to add all of the test*wikis except test2wiki to the environment within the first 1-2 months of operation. Additional wikis are slated to be added later pending establishing compensating controls that will satisfy concerns about code integrity in a fully automated deployment environment. Once completed this will give the environment a mix of wikis with the intent that there is enough organic traffic to the "real" wikis to expose serious defects before code is automatically promoted into the weekly train release process.
A new testing wiki using a right-to-left (RTL) UI language such as Hebrew or Arabic will be added to the pretrain environment following the initial deployment. This RTL wiki will assist test engineers in validating RTL specific functionality and patrolling for regressions. Further discussion is needed to determine the exact UI language and wiki name.
The single version MediaWiki container for pretrain will not require WE6.2.5 routing. Instead traffic can be pointed towards the deployment based on domain. MediaWiki JobQueue will also need special configuration to route jobs to the appropriate wmf/next containers.
All logstash events emitted from MediaWiki Kubernetes containers will be annotated with the OCI container's image and tag (for example "restricted/mediawiki-multiversion:2024-11-07-141556-publish") in the dict of "kubernetes" values attached to the event. Generally we should seek to make log events as self-documenting as possible so that folks do not need deep knowledge of train or pretrain operations to figure out what code and config were active when the log event occurred.
Hypotheses
[edit]If we deploy MediaWiki multiple times per day into a contained area of production we will create the most "production like" environment for staff to use for manual exploratory testing and automated regression checks prior to train deployment to major content wikis.
Implementation will progress via a series of smaller hypotheses which will eventually connect to realize the overarching hypothesis. This approach is being used to avoid attempting to track progress against a single "boil the ocean" hypothesis that could take a year or more to reach an easily measurable state.
The work is currently associated with the 2025-2026 Annual Plan's Wiki Experiences (WE) WE6 objective's WE6.1 key result:
By the end of Q4, the number of train-blocking bugs that make it beyond test wikis is reduced by 10%
The work was associated with the 2024-2025 Annual Plan's Wiki Experiences (WE) WE6 objective's WE6.2 key result:
By the end of Q4, enhance an existing project and perform at least two experiments aimed at providing maintainable, targeted environments moving us towards safe, semi-continuous delivery.
[FY24-25 WE6.2.1] Publish pre-train single version containers
[edit]If we publish a versioned build of MediaWiki, extensions, skins, and Wikimedia configuration at least once per day we will uncover new constraints and establish a baseline of wallclock time needed to perform a build.
Step 0 towards the long term goal of being capable of continuous delivery (CD) into production is being able to deliver faster than the current weekly train process. A daily process would be approximately 3-4 times faster than our current production delivery cadence. We currently envision our eventual capability goal as being able to deliver every 15 minutes. Setting the initial goal two orders of magnitude higher (once per 1440 minutes vs once per 15 minutes) will still expose us to a number of real-world constraints that are not addressed by current workflows. We expect to uncover more details about challenges that will arise in continuing to accelerate the pace of delivery without tipping too quickly into extreme difficulty that could endanger our ability to use an iterative development model.
We are explicitly not constraining where this publishing workflow will be measured at this time. We expect the SRE groups who will need to be involved in deploying into a wikikube environment to be occupied by other goals in the initial months of FY24/25. We are not currently certain what new capabilities would need to be produced to target the current beta cluster shared environment or that doing so would be of long term benefit to the goal, team, or projects. We do expect to deliver beyond a single user development environment and will be able to provide more details as design progresses narrowing the cone of uncertainty for the overall project.
This hypothesis has been declared successfully completed. See our 2024-10-31 progress report for the writeup of what we accomplished and the lessons learned along the way.
[FY24-25 WE6.2.6] Create design document for Pretrain (née Group -1) deployment
[edit]If we gather feedback from QTE, SRE, and individuals with domain specific knowledge and use their feedback to write a design document for deploying and using the wmf/next OCI container, then we will reduce friction when we start deploying that container.
We have a build process that is producing a wmf/next branch and related OCI image daily. That image is not yet being used anywhere however. The next major implementation milestone is to deploy the image into production somewhere with config to serve one or more wikis and edge routing to bring traffic to the deployment. This work will need input from a number of teams and individuals who will have various concerns and constraints that will need to be addressed. Finding consensus on known technical and social questions before attempting to implement the deployment process and its related config should reduce conflicts and confusion for everyone.
This hypothesis has been declared successfully completed. See our 2025-04-04 progress report for the writeup of what we accomplished and the lessons learned along the way.
[FY25-26 WE6.1.1] Move image build to deployment server and update for backports
[edit]If we rehome daily image builds to the deployment server and add image updates triggered by select deployment actions we will uncover constraints and establish a baseline for time needed to perform more continuous deployments.
This is follow up to the work from [FY24-25 WE6.2.1] Publish pre-train single version containers informed by the agreements made during [FY24-25 WE6.2.6] Create design document for Pretrain (née Group -1) deployment. We will migrate the daily image build process from the https://releases-jenkins.wikimedia.org/ service to a systemd timer triggered scap process that happens on the active deployment server. We will also be adding new support in scap for updating the latest wmf/next image with configuration changes and security patches as they are deployed. These actions will prepare us for the "final" Pretrain phase of actually putting the container to work serving content for a set of testing wikis in a future hypothesis.
This hypothesis has been declared successfully completed. See our 2025-09-19 progress report for the writeup of what we accomplished and the lessons learned along the way.
[FY25-26 WE6.1.4] Establish Pretrain production design for MVP
[edit]If we document reliability risks and compensating controls to mitigate those risks, we will create a shared understanding of their implications for the Pretrain productionization design, sufficient to establish consensus on a path forward to enable an MVP.
This is follow up to the agreements made during [FY24-25 WE6.2.6] Create design document for Pretrain (née Group -1) deployment to clarify, document, and attempt to mitigate various risks to the security, performance, and reliability of the Pretrain project. We hope to come out of this phase of the project with a shared understanding of how to implement serving test.wikipedia.org traffic with code from the wmf/next branch maintained by the existing nightly Pretrain process(es) deployed by automation on a known cadence.