Continuous integration/Tutorials/Adding a new release pipeline for MediaWiki

This is an advanced tutorial, mostly aimed at members of the Wikimedia Release Engineering Team, to add a new pipeline to Wikimedia CI for MediaWiki core, extensions, skins, and a few other repos which are branched to the  and   release and production branches.

In the example, we are using the work done for creating the  pipeline (i.e., the release branch for MediaWiki 1.34).

Prerequisites

 * A Wikimedia Gerrit account to be able to submit changes for review
 * The branch for which you are adding CI does not need to exist yet, but this shouldn't be done

Editing Zuul configuration
The configuration is held in repository, in the file.

The file is made of four sections:


 * 1) Pipelines: define event filtering and result handling behavior
 * 2) jobs: finely adjust the behavior on a per job basis
 * 3) project-templates: define common jobs triggering
 * 4) projects: define, for a Gerrit repository, the pipelines to uses and the hierarchy of jobs being triggered.

We will need to adjust each of these:


 * 1) First, define the new pipelines, one for "test" and one for "gate and submit", and define the triggers for them. The test one should go after the previous version's test pipeline:
 * … and similarly, the gate-and-submit one should go after the previous version's gate-and-submit pipeline:
 * 1) Now you need to determine what the test jobs for this version of MediaWiki should be, so you can ensure that they are reflected in your pipelines. For 1.34, the initial target was identical to 1.33, as the supported PHP versions didn't change until later in the development cycle (HHVM, and PHP 7.0 and 7.1 were all dropped from that branch before release). In general, the new branch will follow whatever master has, as new requirements are added to master first.
 * 2) Next, we need to alter each of the project-templates which have MediaWiki-like branches. As of June 2020, these are:
 * There are also commented out references to it in  and   which you should adjust, to avoid later confusion.
 * Take care to ensure that these align with the PHP and other testing environment requirements you have determined above. This the original patch for  following the determination of supported versions, copying the 1.33 values:
 * … and this is the eventual version as of June 2020, supporting only PHP 7.2 and 7.3 (note that the selection of quibble jobs changed as well):
 * 1) Next, you need to define the use of the new pipelines for the direct jobs for MediaWiki itself. Again, take care to ensure that these align with the PHP and other testing environment requirements you have determined above.
 * 2) The last step in layout.yaml is to look for special repos that also get MediaWiki-like branches but don't use any of the templates; as of June 2020, there's just one, VisualEditor/VisualEditor, which you can configure like so:
 * 3) Finally, you need to tell the CI tests themselves about the new pipeline, in , by extending   like so:
 * There are also commented out references to it in  and   which you should adjust, to avoid later confusion.
 * Take care to ensure that these align with the PHP and other testing environment requirements you have determined above. This the original patch for  following the determination of supported versions, copying the 1.33 values:
 * … and this is the eventual version as of June 2020, supporting only PHP 7.2 and 7.3 (note that the selection of quibble jobs changed as well):
 * 1) Next, you need to define the use of the new pipelines for the direct jobs for MediaWiki itself. Again, take care to ensure that these align with the PHP and other testing environment requirements you have determined above.
 * 2) The last step in layout.yaml is to look for special repos that also get MediaWiki-like branches but don't use any of the templates; as of June 2020, there's just one, VisualEditor/VisualEditor, which you can configure like so:
 * 3) Finally, you need to tell the CI tests themselves about the new pipeline, in , by extending   like so:
 * There are also commented out references to it in  and   which you should adjust, to avoid later confusion.
 * Take care to ensure that these align with the PHP and other testing environment requirements you have determined above. This the original patch for  following the determination of supported versions, copying the 1.33 values:
 * … and this is the eventual version as of June 2020, supporting only PHP 7.2 and 7.3 (note that the selection of quibble jobs changed as well):
 * 1) Next, you need to define the use of the new pipelines for the direct jobs for MediaWiki itself. Again, take care to ensure that these align with the PHP and other testing environment requirements you have determined above.
 * 2) The last step in layout.yaml is to look for special repos that also get MediaWiki-like branches but don't use any of the templates; as of June 2020, there's just one, VisualEditor/VisualEditor, which you can configure like so:
 * 3) Finally, you need to tell the CI tests themselves about the new pipeline, in , by extending   like so:
 * 1) The last step in layout.yaml is to look for special repos that also get MediaWiki-like branches but don't use any of the templates; as of June 2020, there's just one, VisualEditor/VisualEditor, which you can configure like so:
 * 2) Finally, you need to tell the CI tests themselves about the new pipeline, in , by extending   like so:
 * 1) Finally, you need to tell the CI tests themselves about the new pipeline, in , by extending   like so:

Save your changes to the files, and send it for review. Your changes will be diffed with the previous configuration, and if you are listed in the CI allow list, your configuration will be automatically validated.

To have it deployed, see the Zuul page; this requires shell access on the continuous server running it.

Note that there may be some JJB changes needed, depending on the CI configuration at the time.

Example patches:

 * Adding 1.34
 * Removing 1.33 (similar process but in reverse, and split for ease of reverting):
 * Remove job and job-templates uses of the pipeline
 * Remove the pipelines themselves