Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally stores configuration of jobs in an XML format. JJB instead maintains jobs as simple descriptions in YAML format, which are then expanded to XML and uploaded to Jenkins through its HTTP API.

Upstream documentation is available at: https://docs.openstack.org/infra/jenkins-job-builder/.

Note: which jobs are triggered for which events is handled by Zuul via.

How it works
All jobs for Wikimedia Jenkins are managed by JJB. To create or change jobs, edit the YAML files in the repository (submit a change, have it reviewed/merged by someone else).

Deployment must be done from the reviewer's local workstation. (Unlike mediawiki deployment, JJB does not run on the server.)

Install JJB
The recommended way is to use a virtualenv defined in, it ensures the proper version of JJB is used. You will need tox, a helper to setup python virtual environments. To install and run jjb:

Configure JJB
JJB requires a basic authentication file with Jenkins credentials. Authentication is only needed if you intent to deploy changes to Jenkins. For local testing and validation, no authentication credentials are needed (just create a dummy file with empty  and   fields).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the  LDAP group to upload jobs.

To obtain an API token, log into Jenkins and open the "Configure" tab of your account (e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Add API Token", give it a name (eg. "jjb"), and then copy the your new API token.

Create a  file within your integration/config.git checkout:

Troubleshooting
A few common problems that may occur when installing JJB.

pyconfig.h
If you get an error message that contains stuff like: ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory #include "pyconfig.h"                      ^ compilation terminated. then you need to install python development dependencies. In Debian / Ubuntu that is: sudo apt-get install python-dev

six.moves
$ jenkins-jobs --conf jenkins_jobs.ini test jjb/ -o output/ Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 6, in    from jenkins_jobs.cmd import main File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in    from six.moves import configparser, StringIO ImportError: No module named six.moves

This happens with the Python 2.7 version that ships with Mac OS X 10.9 (Mavericks). macOS 10.12 and higher are not affected. On the older Mac OS X versions, you can use Homebrew to install a newer Python 2.7.x, and using its  to build jenkins-job-buider fixes the six-moves issue.

Modifying jobs
See also Install JJB - You need to have it installed locally before you can test or validate config files.


 * Modify jobs by editing the YAML files in the  repository.
 * Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format for the Jenkins API, properties might have slightly different names, or might work differently, or might not be supported).
 * Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
 * Once done editing, run the following in your  directory: This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
 * Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Output directory should now contains files like this: $ ls -1 output/ mediawiki-phpunit operations-puppet-validate ...

Deploy changes

 * See also Install JJB - For deploying changes, JJB must be installed, configured, and authenticated

Check out the relevant change for   from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing other changes.

Now use the JJB "test" command to validate the YAML files and generate the XML files that represent the Jenkins jobs.

If that command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!

If there are no errors, review the XML file for your job:

If the job looks correct, upload the updated Jenkins job configurations to Jenkins.

The  command accepts multiple (variadic) arguments, each of which may use the fnmatch patterns.

Debugging JJB

 * You can enable debugging output by passing
 * Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file  to force the update.

Synchronizing all jobs
This might overwrite things you don't intend to. You usually do not want to do this.

Known issues:


 * Updating all jobs does not allow one to control the order in which jobs are uploaded. Given our Jenkins instance is actively used, this will lead to problems where job A depends on job B, but job B is still absent or outdated.

Deploying changes to zuul config
Changes to  are deployed using fabric, by running the following command:

$

See Continuous integration/Zuul for detailed instructions.