Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to automatically generate Jenkins jobs out of YAML files. It generates XML files which are then submitted to Jenkins using its HTTP API. Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

JJB does not setup the Gerrit / Jenkins integration. Whenever jobs for new repositories are created, they MUST first be set up in Zuul.

How it works
All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the  repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB
Wikimedia maintains a fork of JJB in. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone ssh://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Then enable the software using. That will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using.

Configure JJB
The configurations files (YAML) describing the Jenkins jobs are maintained in the  repository. Clone that repository into a directory " " in the working copy of your JJB clone (or put it elsewhere and symlink it):

cd /path/to/jenkins-job-builder git clone -o gerrit ssh://gerrit.wikimedia.org:29418/integration/jenkins-job-builder-config.git config

JJB Authentication
JJB requires a basic authentication file using the INI format. This file holds the location and credentials to a Jenkins installation.

JJB only needs these credentials to be able to update or delete jobs. To simply generate the XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.mediawiki.org/ci/ only allows users in the  LDAP group to modify jobs (granted to Wikimedia Foundation employees and contractors).

The labs installation currently doesn't have authentication set up properly (TODO).

To get your API token, log in and open the "Configure" tab of your account. Click the "Show API Token" button to reveal your API token.

Create :

[jenkins] user=USERNAME password=API_TOKEN url=https://integration.mediawiki.org/ci/

Verify installation
$ cd /path/to/jenkins-job-builder $ mkdir output/ # place where XML files will be written to $ jenkins-jobs test config/ -o output/

should now contains some files (exact list depending on the current configuration):

$ ls -1 output/ mediawiki-core-install-sqlite mediawiki-core-lint mediawiki-core-merge mediawiki-core-phpunit-api mediawiki-core-phpunit-databaseless mediawiki-core-phpunit-misc mediawiki-core-phpunit-parser operations-puppet-validate ...

Each file contains an XML representation of a Jenkins job.

Congratulations!

Modifying jobs
See also Install JJB (you need to have it installed locally in order to verify the files are valid)


 * Modify jobs by editing the YAML files in the  repository.
 * Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
 * Use an editor with YAML syntax highlighting and ideally YAML linting as well. The file format rely on indentation for its structure which often leads to mistakes by people not used to that (similar to how Python works).
 * Once done editing, run the followign: $ cd /path/to/jenkins-job-builder; jenkins-jobs test config/ -o output/ This verified everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
 * Commit your modifications to a local topic branch (keep master clean), and sent to Gerrit for review.

Deploy changes
See also Install JJB (you need to have it installed and authenticated)

Before updating the jobs, run the test command to make sure the YAML files are valid:

$ cd /path/to/jenkins-job-builder $ jenkins-jobs test config/ -o output/

If there any stack trace in that output, your YAML file is invalid. Carefully look at the jobs in output/ and make sure everything is properly setup.

Then push the Job configurations to Jenkins:

$ jenkins-jobs --conf /etc/jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate

If the Labs instance is working properly, push it there first (update the authentication details in the INI file) before deploying in production.

Notes:
 * Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file  to force the update.
 * You can enable debugging output by passing