Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to automatically generate Jenkins jobs out of YAML files. It generates XML files which are then submitted to Jenkins using its HTTP API. Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

JJB does not setup the Gerrit / Jenkins integration. Whenever jobs for new repositories are created, they MUST first be set up in Zuul.

How it works
All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the  repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB
Wikimedia maintains a fork of JJB in. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone ssh://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Then enable the software using. That will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using.

Configure JJB
The configurations files (YAML) describing the Jenkins jobs are maintained in the  repository. Clone that repository into a directory " " in the working copy of your JJB clone (or put it elsewhere and symlink it):

cd /path/to/jenkins-job-builder git clone -o gerrit ssh://gerrit.wikimedia.org:29418/integration/jenkins-job-builder-config.git config

JJB Authentication
JJB requires a basic authentication file using the INI format. This file holds the location and credentials to a Jenkins installation.

JJB only needs these credentials to be able to update or delete jobs. To simply generate the XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.mediawiki.org/ci/ only allows users in the  LDAP group to modify jobs (granted to Wikimedia Foundation employees and contractors).

The labs installation currently doesn't have authentication set up properly (TODO).

To get your API token, log in and open the "Configure" tab of your account. Click the "Show API Token" button to reveal your API token.

Create :

[jenkins] user=USERNAME password=API_TOKEN url=https://integration.mediawiki.org/ci/

Verify installation
$ cd /path/to/jenkins-job-builder $ mkdir output/ # place where XML files will be written to $ jenkins-jobs test config/ -o output/

should now contains some files (exact list depending on the current configuration):

$ ls -1 output/ mediawiki-core-install-sqlite mediawiki-core-lint mediawiki-core-merge mediawiki-core-phpunit-api mediawiki-core-phpunit-databaseless mediawiki-core-phpunit-misc mediawiki-core-phpunit-parser operations-puppet-validate ...

Each file contains an XML representation of a Jenkins job.

Congratulations!

Modifying jobs
Modifications are all about editing the yaml files and asking JJB to update the remote jobs. You definitely want to read the documentation. Also have a look at the existing files.

You will want an editor with YAML syntax highlighting and possibly YAML linting. The file format rely on indentation for its syntax which often leads to mistakes to people not used to that. The python language has a similar paradigm.

Your modifications should be sent to Gerrit for other people to review them, so make sure to test them first using jenkins-jobs test (see below).

Deploy changes
To be able to update the configuration, you will need a username that is able to update jobs using the API and its API key. Wikimedia engineering should be able to use their labs credentials, else one could use jobbuilder-bot account. The configuration is done in jenkins_jobs.ini file:

[jenkins] user=jobbuilder-bot password=0123456789abcdef url=https://integration.mediawiki.org/ci/

Before updating the jobs, run the test command to make sure your .yaml files do not have typos:

jenkins-jobs test config/ -o output/

If there any stack trace in that output, your YAML file is invalid. Then carefully look at the jobs in output/ and make sure everything is properly setup.

Then update the Jenkins instance: $ jenkins-jobs --conf jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate $

Ideally we would want to test the jobs in preproduction before deploying them on production.

notes:
 * Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml to force the update.
 * You can enable debugging output by passing -l debug