Continuous integration/Docker

Jump to navigation Jump to search

As of August 2017, the CI system is experimenting with using Docker containers to run tests.


There is currently no kubernetes infrastructure on which to run containers for testing, nor is there a timeline to create a kubernetes cluster. As such, administrative tasks are handled solely by Jenkins. As a result, our containers should be self-sufficient and tidy. That is, a container should leave behind nothing but logs and rely on nothing but environment variables provided by jenkins (ZUUL_URL, ZUUL_REF, etc).

As of November of 2017, we are working towards moving docker images from docker hub to under the releng namespace.

Jenkins Agent Creation[edit]

  • Create a new instance in horizon with a name following the pattern 'integration-slave-docker-100X'
  • Wait for the first puppet run to complete and log in
  • Run the following to finish switching to the integration puppet master:
sudo rm -fR /var/lib/puppet/ssl
sudo mkdir -p /var/lib/puppet/client/ssl/certs
sudo puppet agent -tv
sudo cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/client/ssl/certs
sudo puppet agent -tv
  • Add the 'role::ci::slave::labs::docker' class to the instance in horizon
    • For larger instance types (m1.xlarge and bigram) specify true for the docker_lvm_volume parameter.
  • Run a final update for puppet 'sudo puppet agent -tv'
  • Pull an initial set of docker images onto the host (using latest tags) to avoid doing this in test runs:
sudo docker pull
sudo docker pull
sudo docker pull
  • Add the slave in the jenkins UI

Image Creation[edit]

Images used in Wikimedia's CI system are created from the integration/config repository. As of Nov 15th 2017 we are in a transition period between using in the integration/config repository, and using docker pkg.

Images using docker-pkg[edit]

docker-pkg is a python3 program that is used to build both CI images and production docker images via Jinja2 templating.

Installing docker-pkg, building docker-pkg images[edit]

  1. Clone the code from docker-pkg and install via pip3
    $ git clone
    $ cd docker-pkg
    $ pip3 install -e .
  2. Clone the integration/config project
    $ git clone
  3. Now you can build all images in the dockerfiles folder using docker-pkg
    $ cd path/to/integration/config
    $ docker-pkg -c dockerfiles/config.yaml dockerfiles

Example output:

== Step 0: scanning dockerfiles ==
Will build the following images:
== Step 1: building images ==
=> Building image
=> Building image
=> Building image
== Step 2: publishing ==
NOT publishing images as we have no auth setup
== Build done! ==
You can see the logs at ./docker-pkg-build.log

Publishing docker-pkg images[edit]

There is a new Fab task called deploy_docker. This task will build and publish all docker-pkg images in integation/config on contint1001

Testing new containers[edit]

Once the new container is pushed to docker hub it should be tested on one of the integration-slave-docker-100x machines. As of August 2017 there are 4 such machines: integration-slave-docker-100[1:4].

To test

  1. ssh to one of the integration-slave-docker machines and su to the jenkins-deploy user.
    you@laptop:~$ ssh integration-slave-docker-1004
    you@integration-slave-docker:~$ sudo su - jenkins-deploy
  2. Create a new directory and an environment file that contains the information passed from Jenkins in the form of ZUUL_* variables
    jenkins-deploy@integration-slave-docker:~$ mkdir docker-test && cd docker-test
    jenkins-deploy@integration-slave-docker:docker-test$ printf "ZUUL_PROJECT=operations/puppet\nZUUL_URL=git://\nZUUL_REF=refs/zuul/production/Ze59ae894f02248d9888835dbaa14dfdf\nZUUL_COMMIT=045fcb14e9fd7885957d900b9a97c883fc5cd26d\n" > .env
  3. Run the new docker container with the environment file and ensure that it runs correctly
    jenkins-deploy@integration-slave-docker:docker-test$ mkdir log
    jenkins-deploy@integration-slave-docker:docker-test$ docker run --rm -it --env-file .env --volume "$(pwd)"/log:/var/lib/jenkins/log contint/operations-puppet
  4. If everything is working as anticipated, update JJB with the Dockerfile version that has been pushed to the Wikimedia Docker registry.