Continuous integration/Tutorials/Test your Python

This tutorial is aimed at getting Jenkins jobs to be triggered for your python project. We will cover steps that needs to be done in your code repository and how to have jobs added and triggered by Wikimedia continuous integration platform.

Design
The jobs are meant to be run using tox, a thin wrapper around virtualenv. As a prerequisite you will need a working setup.py to let tox install your python project in an isolated environment. Each environment defined in the tox file can then be defined as a Jenkins job. Whenever a change is proposed in Gerrit, all the defined jobs will be triggered and report back in Gerrit. The jobs are run on labs instance under the integration labs project.

Prerequisites
You should have a working setup.py installed and a few tests. What is the point of writing a python module if it can not be installed and lacks even the most basic tests. One can use the python module pbr to easily bootstrap a setup.py.

Matching conventions and adding tox support
Your repository should match a bunch of conventions and support tox. Do not be afraid, that is surprisingly very easy to achieve. We will get environments to run the flake8 python linter and nose tests.

Create a configuration file setup.cfg to finely tune the behavior of the utilities:

List your requirements! Most probably your project will require additional modules. The convention in python world is to list the modules in two files:


 * requirements.txt : for modules needed to execute your software
 * test-requirements.txt : additional dependencies when executing tests

The will be added to tox configuration which invokes 'pip install --pre'.

Then create a tox configuration boilerplate. This is done in a file tox.ini at the root of your repository. We will define two environments: py26 and py27 for tests with python 2.6 and 2.7, then flake8 to run the linting utility.

Then you can install tox:

pip install tox

At first list environment triggered by default:

Execute them using tox. If all went well you will seen green success.

You can execute a single environment by passing -e to tox:

Editing Jenkins configuration
Since you have installed Jenkins Job Builder, aim at the configuration directory fetched under config. You will create a job for each tox environment you want triggered using two job templates:


 * {name}-tox-{toxenv} python 2.7.3 or 3.2.3 (Ubuntu Precise)
 * 1) {name}-tox-{toxenv}-trusty</tt> python 2.7.6 or 3.4.0 (Ubuntu Trusty)

The name of the template will be used to forge the name of the Jenkins job. The templates takes two parameters:


 * {name} : a prefix representing your project
 * {toxenv} : the tox environment you want this job to run

Lets say your project is team/software1</tt> and you want to invoke the tox environments py27</tt>, py34 and flake8</tt>. Add your project in your team configuration file (team.yaml), set the list of values for toxenv</tt> and define the job templates to use:

Save the file, you have completed the configuration.

Now run Jenkins jobs builder in testing mode to verify its properly creating the new jobs:

They are written using the XML format Jenkins use to describe jobs.

You can then git add the modified YAML file and submit it for review and later deployment.

See CI/JJB to deploy the change in Jenkins.

Note that team-software1-tox-py34</tt> is not going to work since it is going to run on Ubuntu Precise which does not provide python 3.4. We would simply not trigger it, lets look at how to change Zuul configuration to trigger the proper jobs.

Editing Zuul configuration
The configuration is held in integration/zuul-config.git</tt> repository which has a single file: layout.yaml.

Under the projects</tt> section add an entry for the Gerrit repository name (ex: team/software1</tt>). We will want to trigger on patch proposals and when someone vote +2:


 * 1) team-software1-tox-py27</tt>
 * 2) team-software1-tox-py34-trusty</tt>
 * 3) team-software1-tox-flake8</tt>

Which is defined as:

Save, commit and propose the patch for review.

Refer to Zuul upstream documentation for layout.yaml for more details.