Continuous integration/Jenkins job builder
Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally saves the configuration of jobs in an XML format. JJB outputs the configuration in that format and feeds them to Jenkins through its HTTP API.
Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.
Note: Which jobs are triggered for which events is handled by Zuul.
How it works
All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the integration/config repository (submit a change, have it reviewed/merged by someone else).
To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.
So make sure you have JJB installed. To update it, pull the latest changes from the repository, and then generate the XML files and push them to Jenkins with your credentials.
Wikimedia maintains a fork of JJB in
integration/jenkins-job-builder.git. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.
git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git
Enable the software.
As a prerequisite you will need
python-dev. On Debian based systems you can install them using:
sudo apt-get install python-dev python-pip.
If your python install requires root to install into the Library, run as root (e.g. default Mac OS X)
$ sudo pip install -e .
Or, if your python install only has a writable library for your current user:
$ pip install --user -e .
The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using
jenkins-jobs. (If you try to run
./jenkins_jobs/cmd.py directly it will not generate the XML files.)
After installation., clone the
integration/config repository and symlink
config in the working copy of your JJB install:
$ cd /path/to/integration $ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git $ cd jenkins-job-builder $ ln -s ../config/jjb config # Should now look like this: integration ├── config/ └── jenkins-job-builder/ ├── config -> ../config/jjb
JJB requires a basic authentication file with Jenkins credentials. These are only needed if you need to create, update or delete jobs. To simply validate YAML or generate XML files, you do not need any credentials (just create a dummy file).
Our main installation at https://integration.wikimedia.org/ci/ only allows users in the
wmf LDAP group to modify jobs.
To get your API token, log into Jenkins and open the "Configure" tab of your account (
JENKINS_URL/user/USERNAME/configure; e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Show API Token" button to reveal your API token.
[job_builder] allow_duplicates=True [jenkins] user=USERNAME password=API_TOKEN url=https://integration.wikimedia.org/ci/ query_plugins_info=False
If you want to test a local Jenkins installation, create an .ini file like this:
[job_builder] allow_duplicates=True [jenkins] user= password= url=http://localhost:8080/ query_plugins_info=False
A few common problems that may occur when installing JJB.
If you get an error message that contains stuff like:
ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory #include "pyconfig.h" ^ compilation terminated.
then you need to install python development dependencies. In Ubuntu that is:
sudo apt-get install python-dev
$ jenkins-jobs test config/ -o output/ Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 6, in <module> from jenkins_jobs.cmd import main File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in <module> from six.moves import configparser, StringIO ImportError: No module named six.moves
This happens with the Python 2.7 that ships with Mac OS X 10.9. Installing Python 2.7.8 from Homebrew, and using its
pip to build jenkins-job-buider fixes this.
See also Install JJB (you need to have it installed locally in order to verify the files are valid)
- Modify jobs by editing the YAML files in the
- Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
- Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
- Once done editing, run the following in
$ jenkins-jobs test config/ -o output/This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
- Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.
Output directory should now contains files like this:
$ ls -1 output/ mediawiki-core-phplint mediawiki-core-npm mediawiki-phpunit operations-puppet-validate ...
- Install JJB (you need to have it installed, configured and authenticated)
Check out the relevant integration/config change from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing changes to generic macros that are included in all jobs.
integration/config$ git review -d 139035
Before each deployment, update JJB to latest version by pulling master and re-running the quick installer.
integration/jenkins-job-builder$ git pull origin master # Requires sudo on OSX by default integration/jenkins-job-builder$ sudo pip install -e . # running develop... # Installing pip script to /usr/local/bin # Finished processing dependencies for jenkins-job-builder
Now use the JJB "test" command to validate the YAML files and generate the XML files. If that command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!
integration/jenkins-job-builder$ rm -f output/* && jenkins-jobs test config/ -o output/ .. # Look at the created file integration/jenkins-job-builder$ cat output/example-jenkins-job .. # Deploy one job? integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'example-jenkins-job' # INFO:root:Updating jobs in config/ (['operations-apache-config-lint']) # INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-apache-config-lint # You can also use a wildcard to deploy multiple jobs at once $ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ '*-example'
Synchronizing all jobs
- You should not be doing this
This will easily take over an hour and might overwrite things you don't intend to. It's like doing scap instead of sync-file in Wikimedia's deployment. In addition it doesn't allow you to control the order of things, so you might update jobs with dependencies before the dependencies, which is problematic as we're pushing to a production instance that is actively used.
$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ INFO:root:Updating jobs in config/ (None) INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate ..... INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-merge INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jshint INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser ....
- You can enable debugging output by passing
- Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file
~/.jenkins_jobs_cache.ymlto force the update.