Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally saves the configuration of jobs in an XML format. JJB outputs the configuration in that format and feeds them to Jenkins through its HTTP API.

Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

Note: Which jobs are triggered for which events is handled by Zuul.

How it works
All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB
Wikimedia maintains a fork of JJB in. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Enable the software.

As a prerequisite you will need pip and python-dev. On Debian based systems you can install them using: sudo apt-get install python-dev python-pip.

If your python install requires root to install into the Library, run as root (e.g. default Mac OS X) $ sudo pip install -e. Or, if your python install only has a writable library for your current user: $ pip install --user -e.

The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using. (If you try to run  directly it will not generate the XML files.)

Configure JJB
After installation., clone the  repository and symlink  to   in the working copy of your JJB install:

$ cd /path/to/integration $ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git $ cd jenkins-job-builder $ ln -s ../config/jjb config

integration ├── config/ └── jenkins-job-builder/ ├── config -> ../config/jjb
 * 1) Should now look like this:

Authenticate JJB
JJB requires a basic authentication file with Jenkins credentials. These are only needed if you need to create, update or delete jobs. To simply validate YAML or generate XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the  LDAP group to modify jobs.

To get your API token, log into Jenkins and open the "Configure" tab of your account (e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Show API Token" button to reveal your API token.

Create :

[job_builder] allow_duplicates=True

[jenkins] user=USERNAME password=API_TOKEN url=https://integration.wikimedia.org/ci/

If you want to test a local Jenkins installation, create an .ini file like this:

[job_builder] allow_duplicates=True

[jenkins] user= password= url=http://localhost:8080/

Troubleshooting
A few common problems that may occur when installing JJB.

pyconfig.h
If you get an error message that contains stuff like: ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory #include "pyconfig.h"                      ^ compilation terminated. then you need to install python development dependencies. In Ubuntu that is: sudo apt-get install python-dev

six.moves
$ jenkins-jobs test config/ -o output/ Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 6, in    from jenkins_jobs.cmd import main File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in    from six.moves import configparser, StringIO ImportError: No module named six.moves

This happens with the Python 2.7 that ships with Mac OS X 10.9. Installing Python 2.7.8 from Homebrew, and using its  to build jenkins-job-buider fixes this.

Modifying jobs
See also Install JJB (you need to have it installed locally in order to verify the files are valid)


 * Modify jobs by editing the YAML files in the  repository.
 * Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
 * Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
 * Once done editing, run the following in :  $ jenkins-jobs test config/ -o output/ This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
 * Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Output directory should now contains files like this: $ ls -1 output/ mediawiki-core-phplint mediawiki-core-npm mediawiki-phpunit operations-puppet-validate ...

Deploy changes

 * Install JJB (you need to have it installed, configured and authenticated)

Check out the relevant integration/config change from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing changes to generic macros that are included in all jobs.

Before each deployment, update JJB to latest version by pulling master and re-running the quick installer.

Now use the JJB "test" command to validate the YAML files and generate the XML files. If that command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!

Synchronizing all jobs

 * You should not be doing this

This will easily take over an hour and might overwrite things you don't intend to. It's like doing scap instead of sync-file in Wikimedia's deployment. In addition it doesn't allow you to control the order of things, so you might update jobs with dependencies before the dependencies, which is problematic as we're pushing to a production instance that is actively used.

$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate ..... INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-merge INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jshint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser ....