Continuous integration/Documentation generation

Documentation is automatically generated and published to doc.wikimedia.org. For increased flexibility and security, the static sites are generated inside WMCS instances. The docs then fetched from the instance to the CI production host and then pushed with rsync to the documentation host ( since January 9th 2019). This page documents the workflow, part of the technical implementation, and how to define a new job.

Zuul
Our reacts on two kinds of Gerrit events which are matched by two different pipelines:

could cover the changes being merged since the target branch is updated, but the event is not associated with a Gerrit change number which prevents us from reporting back to the Gerrit interface. We thus use:


 * pipeline to report back in Gerrit so the user knows the documentation has been generated
 * pipeline which solely handles references updates matching.

In both case ( or  ) we trigger the same job to generate the documentation for any branch or tag. We thus need to namespace the documentation under https://doc.wikimedia.org based on the branch name or the tag name. The information is carried differently between events and the reference is slightly different between branch updates and tags. The conversion logic is carried by a which is applied to all the publish jobs. It injects to the Gearman function (and thus the Jenkins job environment) a variable   which represents the version.

Example:


 * A change merged on  branch:
 * updated:

Reference:

We can thus reuse that  parameter to easily namespace the jobs per versions. As an example https://doc.wikimedia.org/mediawiki-core/ has documentation for both release branches (https://doc.wikimedia.org/mediawiki-core/REL1_37/) and tags (https://doc.wikimedia.org/mediawiki-core/1.36.2).

Jenkins job builder definitions
Most of the logic is defined in Jenkins Job Builder  configuration file.

In a job definition, a  defines what steps to execute. We provide a builder macro called  that takes care of transferring the generated files to the web server of https://doc.wikimedia.org/. It takes two parameters:
 * 1)   Directory holding documentation files relative to workspace (without trailing slash)
 * 2)   Directory under https://doc.wikimedia.org/.

Example job definition:

This will invoke the build scripts (doxygen and jsduck) and publish their results (respectively in doc/html and docs) to https://doc.wikimedia.org/myproject/.

To namespace the documentation based on project version, use the Zuul-generated  (derived from branch or tag name). Simply insert it in the  parameter. You will need to invoke the builder assert-env-doc_subpath. Example for MediaWiki (mediawiki-core-doxygen-publish job):

This publishes the documentation at https://doc.wikimedia.org/mediawiki-core/master/php/, and also publishes release branch documentation such as https://doc.wikimedia.org/mediawiki-core/REL1_34/php/, and tagged releases such as https://doc.wikimedia.org/mediawiki-core/1.34.0/php/

Architecture
The documentations are ultimately published on doc.wikimedia.org which is a production machine (doc1001.eqiad.wmnet since January 2019). They are generated on WMCS instance part of the  Toolforge project which are not allowed to communicate with production machines.

A job publish-to-doc1001 executes on the CI Jenkins production node which fetches artifacts from the build workspace using rsync + ssh. The content is then rsynced to doc1001.eqiad.wmnet (rsync://doc1001.eqiad.wmnet/doc/).

Updating the doc.wikimedia.org site
The configuration is done with Puppet by

The "static" content of the site is in the git repository. Most of the pages use light PHP to show a table of content, include a footer, etc.

The git docroot is deployed to the doc host(s) using scap:

ssh deployment.eqiad.wmnet cd /srv/deployment/integration/docroot git fetch origin # review the diff git rebase scap deploy ''