Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally saves the configuration of jobs in an XML format. JJB outputs the configuration in that format and feeds them to Jenkins through its HTTP API.

Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

Note: Which jobs are triggered for which events is handled by Zuul.

How it works
All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB
Wikimedia maintains a fork of JJB in. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Enable the software.

If your python install requires root to install into the Library, run as root (e.g. default Mac OS X) $ sudo pip install -e. Or, if your python install only has a writable library for your current user: $ pip install --user -e.

The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using. (If you try to run  directly it will not generate the XML files.)

What could go wrong
If you get an error message that contains stuff like: ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory #include "pyconfig.h"                      ^ compilation terminated. then you need to install python development dependencies. In Ubuntu that is: sudo apt-get install python-dev

Configure JJB
Clone the repository and symlink to   in the working copy of your JJB clone:

$ cd /path/to/integration $ ls .. jenkins-job-builder .. $ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git $ cd jenkins-job-builder $ ln -s ../config/jjb config

JJB Authentication
JJB requires a basic authentication file using the INI format. This file holds the location and credentials to a Jenkins installation.

JJB only needs these credentials to be able to update or delete jobs. To simply generate the XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the  LDAP group to modify jobs (granted to Wikimedia Foundation employees and contractors).

To get your API token, log in and open the "Configure" tab of your account (currently https://integration.wikimedia.org/ci/user/USERNAME/configure). Click the "Show API Token" button to reveal your API token.

Create :

[job_builder] allow_duplicates=True

[jenkins] user=USERNAME password=API_TOKEN url=https://integration.wikimedia.org/ci/

If you want to test a local Jenkins installation, create an .ini file like this:

[job_builder] allow_duplicates=True

[jenkins] user= password= url=http://localhost:8080/

Verify installation
$ cd /path/to/jenkins-job-builder $ mkdir output/ # place where XML files will be written to $ jenkins-jobs test config/ -o output/

should now contains some files (exact list depending on the current configuration):

$ ls -1 output/ mediawiki-core-install-sqlite mediawiki-core-lint mediawiki-core-merge mediawiki-core-phpunit-api mediawiki-core-phpunit-databaseless mediawiki-core-phpunit-misc mediawiki-core-phpunit-parser operations-puppet-validate ...

Each file contains an XML representation of a Jenkins job.

Congratulations!

Modifying jobs
See also Install JJB (you need to have it installed locally in order to verify the files are valid)


 * Modify jobs by editing the YAML files in the  repository.
 * Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
 * Use an editor with YAML syntax highlighting and ideally YAML linting as well. The file format rely on indentation for its structure which often leads to mistakes by people not used to that (similar to how Python works).
 * Once done editing, run the following in :  $ jenkins-jobs test config/ -o output/ This verified everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
 * Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Deploy changes
See also Install JJB (you need to have it installed and authenticated)

Check out the relevant change from Gerrit into the jenkins-job-builder-config repo. Ensure the change is rebased onto latest master to avoid accidentally undoing change to generic macros that are included in all jobs.


 * 1) Before letting JJB parse the YAML files into XML. Ensure sure you are running the latest version of JJB by pulling master and re-running the quick installer.
 * 2) Running the   command will validate the YAML files and generate the XML files in   directory. If that commands results in error, there is either a YAML syntax error or incorrect JJB commands inside the YAML file. Resolve this before continuing!
 * 3) Push the updated job configuration to Jenkins.

Example

 * 1) Checkout config change

krinkle in integration/config (master) $ git review -d 139035 Downloading refs/changes/35/139035/3 from gerrit


 * 1) Ensure jenkins-job-builder is installed and up to date

krinkle in integration/config (review/ori_livneh/139035) $ cd ../jenkins-job-builder

krinkle at integration/jenkins-job-builder (master) $ ls -l config config@ -> ../config/jjb

krinkle in integration/jenkins-job-builder (master) $ git pull origin master From ssh://gerrit.wikimedia.org:29418/integration/jenkins-job-builder Already up-to-date.

krinkle in integration/jenkins-job-builder (master) $ pip install -e. running develop... Installing pip script to /usr/local/bin Finished processing dependencies for jenkins-job-builder


 * 1) Compile job XML files

krinkle in integration/jenkins-job-builder (master) $ rm output/* && jenkins-jobs test config/ -o output/

krinkle in integration/jenkins-job-builder (master) $ cat output/operations-apache-config-lint ...


 * 1) Push one or more jobs configurations to Jenkins (updates or creates the given job)

krinkle in integration/jenkins-job-builder (master) $ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'operations-apache-config-lint' INFO:root:Updating jobs in config/ (['operations-apache-config-lint']) INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-apache-config-lint

Synchronizing all jobs

 * You should not be doing this

This will easily take over an hour and might overwrite things you don't intend to. It's like doing scap instead of sync-file in Wikimedia's deployment. In addition it doesn't allow you to control the order of things, so you might update jobs with dependencies before the dependencies, which is problematic as we're pushing to a production instance that is actively used.

$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate ..... INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-merge INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jshint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser ....