Continuous integration/Jenkins job builder

From MediaWiki.org
Jump to: navigation, search
shortcut: CI/JJB

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally saves the configuration of jobs in an XML format. JJB outputs the configuration in that format and feeds them to Jenkins through its HTTP API.

Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

Note: Which jobs are triggered for which events is handled by Zuul.

How it works[edit]

All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the integration/config repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB[edit]

Wikimedia maintains a fork of JJB in integration/jenkins-job-builder.git. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Enable the software.

As a prerequisite you will need pip and python-dev. On Debian based systems you can install them using: sudo apt-get install python-dev python-pip.

If your python install requires root to install into the Library, run as root (e.g. default Mac OS X)

$ sudo pip install -e .

Or, if your python install only has a writable library for your current user:

$ pip install --user -e .

The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using jenkins-jobs[1]. (If you try to run ./jenkins_jobs/cmd.py directly it will not generate the XML files.)

Configure JJB[edit]

After installation., clone the integration/config repository and symlink config/jjb[2] to config in the working copy of your JJB install:

$ cd /path/to/integration
$ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git
$ cd jenkins-job-builder
$ ln -s ../config/jjb config

# Should now look like this:
integration
├── config/
└── jenkins-job-builder/
    ├── config -> ../config/jjb

Authenticate JJB[edit]

JJB requires a basic authentication file with Jenkins credentials. These are only needed if you need to create, update or delete jobs. To simply validate YAML or generate XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the wmf LDAP group to modify jobs.

To get your API token, log into Jenkins and open the "Configure" tab of your account (JENKINS_URL/user/USERNAME/configure; e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Show API Token" button to reveal your API token.

Create /path/to/jenkins-job-builder/etc/jenkins_jobs.ini:

[job_builder]
allow_duplicates=True

[jenkins]
user=USERNAME
password=API_TOKEN
url=https://integration.wikimedia.org/ci/
query_plugins_info=False

If you want to test a local Jenkins installation, create an .ini file like this:

[job_builder]
allow_duplicates=True

[jenkins]
user=
password=
url=http://localhost:8080/
query_plugins_info=False

Troubleshooting[edit]

A few common problems that may occur when installing JJB.

pyconfig.h[edit]

If you get an error message that contains stuff like:

 ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory
  #include "pyconfig.h"
                      ^
 compilation terminated.

then you need to install python development dependencies. In Ubuntu that is:

sudo apt-get install python-dev

six.moves[edit]

$ jenkins-jobs test config/ -o output/
Traceback (most recent call last):
  File "/usr/local/bin/jenkins-jobs", line 6, in <module>
    from jenkins_jobs.cmd import main
  File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in <module>
    from six.moves import configparser, StringIO
ImportError: No module named six.moves

This happens with the Python 2.7 that ships with Mac OS X 10.9. Installing Python 2.7.8 from Homebrew, and using its pip to build jenkins-job-buider fixes this.

Modifying jobs[edit]

See also Install JJB (you need to have it installed locally in order to verify the files are valid)

  • Modify jobs by editing the YAML files in the integration/config.git repository.
  • Read the documentation[2] to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
  • Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
  • Once done editing, run the following in /path/to/jenkins-job-builder:
    $ jenkins-jobs test config/ -o output/
    
    This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
  • Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Output directory should now contains files like this:

$ ls -1 output/
mediawiki-core-phplint
mediawiki-core-npm
mediawiki-phpunit
operations-puppet-validate
...

Deploy changes[edit]

Install JJB (you need to have it installed, configured and authenticated)

Check out the relevant integration/config change from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing changes to generic macros that are included in all jobs.

integration/config$ git review -d 139035

Before each deployment, update JJB to latest version by pulling master and re-running the quick installer.

integration/jenkins-job-builder$ git pull origin master

# Requires sudo on OSX by default
integration/jenkins-job-builder$ sudo pip install -e .
 # running develop...
 # Installing pip script to /usr/local/bin
 # Finished processing dependencies for jenkins-job-builder

Now use the JJB "test" command to validate the YAML files and generate the XML files. If that command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!

integration/jenkins-job-builder$ rm -f output/* && jenkins-jobs test config/ -o output/
 ..

# Look at the created file
integration/jenkins-job-builder$ cat output/example-jenkins-job
 ..

# Deploy one job?
integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'example-jenkins-job'

 # INFO:root:Updating jobs in config/ (['operations-apache-config-lint'])
 # INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-apache-config-lint

# You can also use a wildcard to deploy multiple jobs at once
$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ '*-example'

Synchronizing all jobs[edit]

You should not be doing this

This will easily take over an hour and might overwrite things you don't intend to. It's like doing scap instead of sync-file in Wikimedia's deployment. In addition it doesn't allow you to control the order of things, so you might update jobs with dependencies before the dependencies, which is problematic as we're pushing to a production instance that is actively used.

$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/
INFO:root:Updating jobs in config/ (None)
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate
.....
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-merge
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jshint
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser
....

Notes[edit]

  • You can enable debugging output by passing -l debug
  • Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml to force the update.

References[edit]