Continuous integration/Docker

As of August 2017, the CI system is experimenting with using Docker containers to run tests.

Overview
There is currently no kubernetes infrastructure on which to run containers for testing, nor is there a timeline to create a kubernetes cluster. As such, administrative tasks are handled solely by Jenkins. As a result, our containers should be self-sufficient and tidy. That is, a container should leave behind nothing but logs and rely on nothing but environment variables provided by jenkins (, , etc).

As of November of 2017, we are working towards moving docker images from docker hub to  under the   namespace.

Jenkins Agent Creation
sudo rm -fR /var/lib/puppet/ssl sudo mkdir -p /var/lib/puppet/client/ssl/certs sudo puppet agent -tv sudo cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/client/ssl/certs sudo puppet agent -tv sudo docker pull docker-registry.wikimedia.org/releng/castor:latest sudo docker pull docker-registry.wikimedia.org/releng/quibble-stretch:latest sudo docker pull docker-registry.wikimedia.org/wikimedia-stretch:latest
 * Create a new instance in horizon with a name following the pattern 'integration-slave-docker-100X'
 * Wait for the first puppet run to complete and log in
 * Run the following to finish switching to the integration puppet master:
 * Add the 'role::ci::slave::labs::docker' class to the instance in horizon
 * For larger instance types ( and  ) specify   for the   parameter.
 * Run a final update for puppet 'sudo puppet agent -tv'
 * Pull an initial set of docker images onto the host (using latest tags) to avoid doing this in test runs:
 * Add the slave in the jenkins UI

Image Creation
Images used in Wikimedia's CI system are created from the integration/config repository. As of Nov 15th 2017 we are in a transition period between using  in the integration/config repository, and using docker pkg.

Images using
is a python3 program that is used to build both CI images and production docker images via Jinja2 templating.

Installing, building docker-pkg images

 * 1) Clone the code from   and install via pip3
 * 2) Clone the   project
 * 3) Now you can build all images in the   folder using

Example output:

Step 0: scanning dockerfiles
Will build the following images:
 * docker-registry.wikimedia.org/releng/ci-stretch:0.1.0
 * docker-registry.wikimedia.org/releng/operations-puppet:0.1.0
 * docker-registry.wikimedia.org/releng/ci-jessie:0.3.0

Step 1: building images
=> Building image docker-registry.wikimedia.org/releng/ci-stretch:0.1.0 => Building image docker-registry.wikimedia.org/releng/operations-puppet:0.1.0 => Building image docker-registry.wikimedia.org/releng/ci-jessie:0.3.0

Step 2: publishing
NOT publishing images as we have no auth setup

Build done!
You can see the logs at ./docker-pkg-build.log

Publishing images
There is a new Fab task called. This task will build and publish all  images in   on

Manage local images
List local images: $ docker images Remove local images from wikimedia.org (source): $ docker rmi $(docker images --format ' :  ' | grep 'wikimedia.org') Fetch images from wikimedia.org to local (source): $ cd integration/config $ ack -o -h -s 'docker-registry.*:[.\d]+' jjb/ | sort | uniq | xargs -n1 docker pull

Test a container locally
Use the below steps to test a docker image locally. This can be unpublished image you've built locally with, or one that was pulled from the wikimedia.org repository.

Note that the below uses urls for the names of the images, but these refer to the ones you have locally (either created or pulled), they do not need to have been deployed or uploaded there yet. You can list the images you have locally using the  command. $ cd my-gerrit-project $ mkdir -m 777 cache log $ docker run \ --rm --tty \ --volume /"$(pwd)"/log://var/lib/jenkins/log \ --volume /"$(pwd)"/cache://cache \ --volume /"$(pwd)"://src \ docker-registry.wikimedia.org/releng/node10-test:0.3.0

Debug a container locally
The default behaviour for  is to start the container and execute the entrypoint/cmd specified in the Dockerfile. To inspect the container instead, specify  to make it interactive, and override   to a shell (such as  ). For example: $ mkdir -m 777 cache log $ docker run \ --rm --tty \ --volume /"$(pwd)"/log://var/lib/jenkins/log \ --volume /"$(pwd)"/cache://cache \ --volume /"$(pwd)"://src \ -i --entrypoint /bin/bash docker-registry.wikimedia.org/releng/node10-test:0.3.0 nobody@5f4cdb0ab167:/src$ nobody@5f4cdb0ab167:/src$ env LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 CHROMIUM_FLAGS=--no-sandbox PWD=/src HOME=/nonexistent NPM_CONFIG_CACHE=/cache XDG_CACHE_HOME=/cache PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Test a container in CI
Once the new container is pushed to docker hub it should be tested on one of the  machines. As of August 2017 there are 4 such machines:.

To test


 * 1) ssh to one of the   machines and   to the   user.
 * 2) Create a new directory and an environment file that contains the information passed from Jenkins in the form of   variables
 * 3) Run the new docker container with the environment file and ensure that it runs correctly
 * 4) If everything is working as anticipated, update JJB with the Dockerfile version that has been pushed to the Wikimedia Docker registry.