Continuous integration/Jenkins job builder

From MediaWiki.org
Jump to navigation Jump to search
shortcut: CI/JJB

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally stores configuration of jobs in an XML format. JJB instead maintains jobs as simple descriptions in YAML format, which are then expanded to XML and uploaded to Jenkins through its HTTP API.

Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

Note: Which jobs are triggered for which events is handled by Zuul.

How it works[edit]

All jobs for Wikimedia Jenkins are managed by JJB. To create or change jobs, edit the YAML files in the integration/config repository (submit a change, have it reviewed/merged by someone else).

Deployment must be done from the reviewer's local workstation. (Unlike mediawiki deployment, JJB does not run on the server.)

Install JJB[edit]

Wikimedia maintains a fork of JJB in integration/jenkins-job-builder.git. This allows us to use and test our own patches (if needed) before having them reviewed upstream. It also ensures we keep our Jenkins and JJB version compatibility in sync.

To install JJB, first clone the following repository.

git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

As a prerequisite you will need pip and python-dev. On Debian-based systems you can install them using: sudo apt-get install python-dev python-pip.

pip install -e .

If your python install requires root (eg. as is the case on macOS by default), run the command with sudo instead:

$ sudo pip install -e .

Or, if your python install only has a writable library for your current user, use --user:

$ pip install --user -e .

The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using the jenkins-jobs command[1].

Configure JJB[edit]

After installation, also clone the integration/config.git repository from WMF and symlink config/jjb[2] to config in the working copy of your JJB install:

$ cd /path/to/git-repos
$ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git integration-config
$ cd jenkins-job-builder
$ ln -s ../integration-config/jjb config

# On Windows you will need to do the following command in cmd:
# "mklink /D config ..\config\jjb"
# Many 'ln' implementations on Windows don't create the symbolic link correctly.

# Should now look like this:
git-repos
├── integration-config/
└── jenkins-job-builder/
    ├── config -> ../integration-config/jjb

Authenticate JJB[edit]

JJB requires a basic authentication file with Jenkins credentials. Authentication is only needed if you intent to deploy changes to Jenkins. For local testing and validation, no authentication credentials are needed (just create a dummy file with empty user and password fields).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the wmf LDAP group to upload jobs.

To obtain an API token, log into Jenkins and open the "Configure" tab of your account (JENKINS_URL/user/USERNAME/configure; e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Show API Token" button to reveal your API token.

Create a etc/jenkins_jobs.ini file within your jenkins-job-builder checkout:

[job_builder]
allow_duplicates=True

[jenkins]
user=USERNAME
password=API_TOKEN
url=https://integration.wikimedia.org/ci/
query_plugins_info=False

If you want to test a local Jenkins installation, set url to where your installation, eg. http://localhost:8080/

Troubleshooting[edit]

A few common problems that may occur when installing JJB.

pyconfig.h[edit]

If you get an error message that contains stuff like:

 ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory
  #include "pyconfig.h"
                      ^
 compilation terminated.

then you need to install python development dependencies. In Ubuntu that is:

sudo apt-get install python-dev

six.moves[edit]

$ jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/
Traceback (most recent call last):
  File "/usr/local/bin/jenkins-jobs", line 6, in <module>
    from jenkins_jobs.cmd import main
  File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in <module>
    from six.moves import configparser, StringIO
ImportError: No module named six.moves

This happens with the Python 2.7 version that ships with Mac OS X 10.9 (Mavericks). macOS 10.12 and higher are not affected. On the older Mac OS X versions, you can use Homebrew to install a newer Python 2.7.x, and using its pip to build jenkins-job-buider fixes the six-moves issue.

Modifying jobs[edit]

See also Install JJB - You need to have it installed locally before you can test or validate config files.

  • Modify jobs by editing the YAML files in the integration/config.git repository.
  • Read the documentation[2] to learn what features are available (JJB internally maps the YAML format to the XML format for the Jenkins API, properties might have slightly different names, or might work differently, or might not be supported).
  • Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
  • Once done editing, run the following in your jenkins-job-builder directory:
    $ jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/
    This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
  • Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Output directory should now contains files like this:

$ ls -1 output/
mediawiki-phpunit
operations-puppet-validate
...

Deploy changes[edit]

See also Install JJB - For deploying changes, JJB must be installed, configured, and authenticated

Check out the relevant change for integration/config from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing other changes.

integration/config$ git review -d 139035

Before each deployment, ensure your JJB installation is up to date by by pulling origin/master and re-running the quick installer.

integration/jenkins-job-builder$ git pull origin master

# Example for macOS, which requires 'sudo' by default
integration/jenkins-job-builder$ sudo pip install -e .
 # ...
 # Installing pip script to /usr/local/bin
 # Finished processing dependencies for jenkins-job-builder

Now use the JJB "test" command to validate the YAML files and generate the XML output. If that command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!

integration/jenkins-job-builder$ rm -f output/* && jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/
 ..

# Look at the created file
integration/jenkins-job-builder$ cat output/example-jenkins-job
 ..

# Deploy one job?
integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'example-jenkins-job'

 # INFO:root:Updating jobs in config/ (['operations-apache-config-lint'])
 # INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-apache-config-lint

# You can also use a wildcard to deploy multiple jobs at once
$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ '*-example'

Notes[edit]

  • You can enable debugging output by passing -l debug
  • Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml to force the update.

Synchronizing all jobs[edit]

This will easily take over an hour and might overwrite things you don't intend to. Unless there has been an outage and most jobs are jobs or corrupted, or you have another very good reason, do not do this. Known issues:

  • Updating all jobs does not allow to control the order in which jobs are uploaded. Given our Jenkins instance is actively used, this will lead to problems where job A depends on job B, but job B is still absent or outdated

Even if you need to deploy changes to many jobs at once, use the one-job or wild-card pattern described above instead, and make sure to upload things in the correct order.

$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/
INFO:root:Updating jobs in config/ (None)
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate
.....
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit
....

References[edit]