Continuous integration/Jenkins job builder

Jenkins Job builder, often abbreviated JJB, is a python script to automatically generate Jenkins jobs out of .yaml files describing them. It generates the .xml files and then update Jenkins installation using its Json API. Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

JJB does not setup the Gerrit / Jenkins integration. Whenever new jobs are created, they MUST also be configured Zuul which is the Gerrit / Jenkins gateway.

Getting code
Wikimedia maintains its own fork of JJB under integration/jenkins-job-builder.git. This has been made to let us push changes in production before having them reviewed upstream. To install JJB on your computer simply clone from the WMF repository:

git clone ssh://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Then enable the software using python setup.py develop. That will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using jenkins-jobs.

WMF configuration
The jobs configurations files are hosted under integration/jenkins-job-builder-config.git and contains the yaml files describing our Jenkins jobs. You will want to clone that repository using the name config under your JJB working directory:

cd /path/to/jenkins-job-builder git clone -o gerrit ssh://gerrit.wikimedia.org:29418/integration/jenkins-job-builder-config.git config

Configuration
JJB requires a basic configuration file using the .ini format. The file holds the credentials to interact with Jenkins as well as the URL to the Jenkins installation you want to interact with.

If you simply want to generate the XML files, you do not need any credentials but still need a dummy configuration file. To be able to update or delete jobs in Jenkins you must be part of the wmf LDAP group which most often means you are a Wikimedia Foundation employee or a contractor.

The credentials are your labs user account and your Jenkins API token, you can get it by browsing to  https://integration.mediawiki.org/ci/user/USERNAME/configure  (replacing USERNAME with your labs account. Once there, click the [Show API Token] button to reveal your API key.

Then create a file named jenkins_jobs.ini:

[jenkins] user=USERNAME password=API_TOKEN url=https://integration.mediawiki.org/ci/

Replacing above USERNAME with your labs account name and API_TOKEN with your credentials.

Verify your installation
In your jenkins job builder working directory:

mkdir output # place where XML files will be written to jenkins-jobs test config/ -o output/

output/</tt> should now contains some files (exact list depending on the current configuration fetched under config/</tt>):

$ ls -1 output/ mediawiki-core-install-sqlite mediawiki-core-lint mediawiki-core-merge mediawiki-core-phpunit-api mediawiki-core-phpunit-databaseless mediawiki-core-phpunit-misc mediawiki-core-phpunit-parser operations-puppet-validate $

Each file contains the XML configuration of a Jenkins job.

Congratulations.

Modifying Jenkins jobs
Modifications are all about editing the yaml files and asking JJB to update the remote jobs. You definitely want to read the documentation at http://ci.openstack.org/jenkins-job-builder/configuration.html. Also have a look at the existing files.

You will want an editor with YAML syntax highlighting and possibly YAML linting. The file format rely on indentation for its syntax which often leads to mistakes to people not used to that. The python language has a similar paradigm.

Your modifications should be sent to Gerrit for other people to review them, so make sure to test them first using jenkins-jobs test</tt> (see below).

Refreshing Jenkins
To be able to update the configuration, you will need a username that is able to update jobs using the API and its API key. Wikimedia engineering should be able to use their labs credentials, else one could use jobbuilder-bot account. The configuration is done in jenkins_jobs.ini file:

[jenkins] user=jobbuilder-bot password=0123456789abcdef url=https://integration.mediawiki.org/ci/

Before updating the jobs, run the test command to make sure your .yaml files do not have typos:

jenkins-jobs test config/ -o output/

If there any stack trace in that output, your YAML file is invalid. Then carefully look at the jobs in output/ and make sure everything is properly setup.

Then update the Jenkins instance: $ jenkins-jobs --conf jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate $

Ideally we would want to test the jobs in preproduction before deploying them on production.

notes:
 * Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml</tt> to force the update.
 * You can enable debugging output by passing -l debug</tt>