Requests for comment/Extensions continuous integration

Background
We proposed to have some third libraries shipped along MediaWiki core. For technical reasons they are held in an independent repository mediawiki/vendor.git. From a continuous integration point of view, that is a bit challenging though. When someone propose a patch against the master branch of core we can just clone the master branch of the vendor and run tests against a consistent code base. But how will we handle a patch sent to REL1_25 or some wmf branches? We definitely need both repositories to use the same branch.

We also had the long standing issue of patches proposed on extensions release branch having tests run with the master branch of MediaWiki core. That caused issues whenever the MediaWiki core API changed between the targeted release and master branches.

Both points have been solved during Summer 2014 and we would want to extend it to other MediaWiki related repositories such as extensions. We will first present how Zuul establish states of repositories for a given patchset, then the utility that makes it trivial to reproduce such a state on a Jenkins slave taking for example the mediawiki/core and mediawiki/vendor tight integration that is being used today. Finally we will propose to extend such system to all MediaWiki extensions deployed on the Wikimedia cluster.

Zuul behavior
When a change is proposed for merging (by voting Code-Review+2), Zuul enters it in a shared queue and creates a unique git reference for the change. For each subsequent change, an additional reference is created for the changes ahead in the queue which let you speculate the future state of the repositories before the change ahead are effectively merged. If you were to propose two patchset in a row for mediawiki/core, Zuul assumes the first patch is going to pass the tests and land in the branch, thus for the second patchset, it crafts a merge commit of the tip of the branch with a merge of the first patch:

Change1: mediawiki/core + Change 1 Change2: mediawiki/core + Change 1 + Change 2

Zuul then triggers jobs for both states, which thus run in parallel. On success, both changes are merged in the order they entered the queue. If the test for Change 1 ends up failing, Zuul dequeue Change 1 and re-enqueue Change 2 against the tip of the branch (ie without the merge of Change1) and retrigger tests.

Changes are considered to be sharing the same merge queue whenever their trigger definitions share a common job. In september we added a specific job 'mediawiki-gate' to all MediaWiki repositories to make sure they end up being tested as if Change ahead have been already merged. That let us ensure that mediawiki/core and mediawiki/vendor are being tested properly.

The mediawiki/core and mediawiki/vendor repositories share a job mediawiki-vendor-integration, thus when voting +2 on two changes made against those repositories, Zuul detects they are coupled and enter them serially in the same queue. If you were to +2 Change 1 against mediawiki/core @REL1_25 then Change 2 against mediawiki/vendor @REL1_25, Zuul would craft the following git references:

When the first change against mediawiki/core enters the queue:

mediawiki/core   zuul/refs/REL1_25/Z1  (core @REL1_25 + Change 1)

Then for the second change against mediawiki/vendor:

mediawiki/core   zuul/refs/REL1_25/Z2  (core @REL1_25 + Change 1) mediawiki/vendor zuul/refs/REL1_25/Z2  (vendor @REL1_25 + Change 2)

Since the second change is behind a change that modify mediawiki/core, Zuul mark the future state of mediawiki/core with Change 1 applied since that is what will ultimately be the new state of the repository.

Zuul then triggers jobs for each Change by passing the project named for the change and the zuul reference. When testing change 2, the Jenkins job cloneis both repositories and fetch the reference zuul/refs/REL1_25/Z2 on both repositories.

This behavior has been documented by Antoine Musso in upstream documentation as http://ci.openstack.org/zuul/gating.html#cross-projects-dependencies which you might want to read as well.

Reproducing a Zuul state
Cloning multiple repositories and checking out the proper Zuul reference is not straightforward. We have to take in consideration that other repositories might not have the targeted branch and support falling back to a default branch. OpenStack is using a custom set of shell scripts to handle but it had lot of hardcoded expectations that would not match our use case.

Introducing Zuul cloner! Antoine wrote a python script that makes it easy to clone repositories matching Zuul crafted state. It has been upstreamed in Zuul itself and already enhanced by them. In short, the script takes a list of Gerrit project to clone and then attempts to make each repository match the Zuul reference for the proposed patchset, falling back to the branch of the proposed patchset or ultimately the 'master' branch.

For a patch sent to mediawiki/core @REL1_25, the Jenkins job receives Zuul parameters (project that triggered the change, Zuul reference) as environment variable and run:

$ zuul-cloner mediawiki/core mediawiki/vendor

Creating repo mediawiki/core Fetched ref refs/zuul/master/Z123456 from mediawiki/core

Creating repo mediawiki/vendor mediawiki/vendor in Zuul does not have ref refs/zuul/master/Z123456 Falling back to branch master

Then runs the tests.

Examples runs can be found in the mediawiki-vendor-integration job.

To bring in mediawiki/vendor in extensions tests, this system has been applied on all extensions jobs (mwext-*-testextension) to ensure we use the proper core and vendor states.

For extensions having dependencies (such as Flow requiring Mantle), that let us ensure a Flow change is tested with a Mantle change ahead in the queue. Such dependencies are manually defined in the Jenkins jobs and do not define all potential interactions we can between all our extensions. We need a better plan.

Testing extensions together
The Wikimedia cluster has 155 extensions and 5 skin. Any change made to one of the repositories can potentially impacts the others and we do not tests that. A change proposed for a skin will not trigger the tests of the other extensions such as the qunit jobs, nor would a change to Mantle run the test of extensions that rely on it (such as Flow). To achieve that, we would need a single job triggered by all the repositories and running all the tests. Thus changes will share the same gating queue, ensuring no change ends up failling tests of another repo.

Antoine had that idea back in December 2013 (  common gating job for mediawiki core and extensions) which led to the creation of the zuul cloner utility during the summer of 2014.

Making tests pass together
While Ori Livneh prepared the migration to HHVM, he crafted a MediaWiki Vagrant role to easily clone all extensions and run their tests together. That highlighted a lot of issues such as:


 * not supporting SQLite as a backend
 * lack of a default PHP entry point such as MyExtension/MyExtension.php
 * extensions registering hooks that change core behavior causing other extensions to fail (ex: thumbnailBeforeProduceHTML
 * tests not passing at all

Most are tracked by Bug 67216 - Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM. Only a few extensions are problematics though and could be ignored while they are being worked on.

Selecting extensions
We have a few challenges to solve to establish a list of repositories (extensions, skins) to be cloned when a patch is proposed for a given branch. A rather incomplete list:


 * we can not assume all repositories have the branch of the proposed patchset. Wikidata and Wikimedia fundraising uses a 'production' branch which is updated outside of the main wmf release train. Zuul cloner supports specifying a fallback branch over than the master one though.
 * In the previous section, we have seen some extensions badly interacts with other because they change code behavior, hence we will need a way to blacklist them.
 * On the master branch of MediaWiki core, we might want to introduce other extensions not installed on the Wikimedia cluster.
 * MediaWiki wmf branch defines dependencies via submodules, which might be ahead of the wmf branches in the original repositories. Thus we might end up testing an outdated branch instead of the one referenced as a submodule.

Selecting extensions
Our mediawiki/tools/release.git repository has utilities that carry some dependencies informations: make-wmf-branch and make-release.


 * make-wmf-branch

The defines what will be included in the future wmf branch. It contains a list of extensions and skins ($branchedExtensions and $branchedSkins) to cut branch for and a list of repositories for which we want an explicit branch ($specialExtensions).

We could make that extension to have release and wmf branches as well. The Jenkins job would thus clone it first and checkout the proposed patchset branch of the repository which would give us the expected state. From there we can craft the list of repositories to be cloned by Zuul cloner with project specific branches as needed.

Antoine experimented with such a script and proposed it as. When run, it inspects the default.conf file and craft a list of parameters suitable for usage by zuul-cloner. Hence a Jenkins job triggered by a patch on the REL1_25 branch would:

$ zuul-cloner mediawiki/tools/release Creating repo mediawiki/tools/release Checked out REL1_25 $ zuul-cloner $(mediawiki/tools/release/zuulparams) Creating repo mediawiki/core Checked out REL1_25 Creating repo mediawiki/extensions/Foo ...

An other interesting use cases are the bundles already being produced continuously. We might well craft integration jobs to ensure the extensions in those bundles works fine together. Antoine is aware of at least two examples:


 * MediaWiki Language Extension Bundle which ships at least Universal Language Selector, Localisation Updated, Clean Changes, Translate
 * Extension:Semantic_MediaWiki and its related extensions


 * make-release

The file for now defines a single bundle for Semantic MediaWiki. Definining the language extension bundle would be as easy as defining a new entry listing the extensions. Additionally, the team in charge of MediaWiki core releases has interest in defining such bundles for the different supported MediaWiki versions..

A Jenkins job could clone the repository first, inspect the make-release.yaml fine and then trigger a run of zuul cloner to clone them all and test the bundle.

Implementation / migration
With dependencies figured out, we can remove the mwext-*-testextension job and replace them with a new job which fetches dependencies (similar to the existing mediawiki-vendor-integration one, but on steroid). The branch cut before summer 2014 (REL1_22 and REL1_23) will need a lot of patches to be backported to. The other branches and master will definitely need a lot of effort before

Long term possibility
Possibly, knowing a set of HEAD passes together, we could add them all as submodules in a dedicated git repository. Each commit in that repository would thus be known to pass together and hence we have a good indication that the can be deployed on a production system. If we ever attain such a nirvana, we could most probably stop using wmf branches entirely in favor of deploying from an integrated repository. That will avoid us to have to cherry pick changes to the wmf branch since the continuous integration system would have handled it for us automatically.

In Wikimedia context, we would reuse a similar context to run tests for the operations/mediawiki-config.git configuration changes against the deployed version and the head of the integrated repository.