Continuous integration/Documentation generation

We automatically generate documentation on https://doc.wikimedia.org/ which is hosted on a production machine. Since the doc is often generated on labs instance, we had to set up a two step process to be able to move the doc from labs to production. This page document the workflow being used and part of the technical implementation.

Jenkins job builder definitions
Most of the logic is defined straight in Jenkins Job Builder doc.yaml configuration file.

In a job definition, a builder define the command to generate the documentation which ends up being written in a build_path. We then have a macro push-doc which takes care of publishing the documentation to the production server. It takes two parameters:
 * 1) docsrc which is the place where the doc has been generated (build_path from above)
 * 2) docdest the final destination path under https://doc.wikimedia.org/

Example job definition:

Will invoke make doc and publish the content under doc/build/html</tt> at https://doc.wikimedia.org/myproject/.

Architecture
The documentations are ultimately published on doc.wikimedia.org which is a production machine (gallium</tt> as of Sep. 2014). They are generated on labs instance part of the integration</tt> labs project which are not allowed to communicate with production machines.

To solve this, we created an intermediary instance integration-publisher.eqiad.wmflabs</tt>, the push-doc</tt> macro running on the labs instance will rsync the generated content to the instance under the doc</tt> rsync container in a uniquely named sub directory (reusing ZUUL_UUID). The macro then triggers the publish-doc</tt> job with the unique identifier, it will rsync from the intermediary instance to the production machine, thus publishing the doc.

The integration-publisher.eqiad.wmflabs</tt> rsync daemon is reachable by other integration labs instances since they are in the same project. The production slave gallium is allowed connection since it has a public IP and can reach labs, the other production slave lanthanum has a private IP and thus can not reach labs per policy. Hence all jobs should be tied to the contintLabsSlave</tt> label.