Continuous integration/Docker

From MediaWiki.org
Jump to navigation Jump to search

As of August 2017, the CI system is experimenting with using Docker containers to run tests.

Overview[edit]

There is currently no kubernetes infrastructure on which to run containers for testing, nor is there a timeline to create a kubernetes cluster. As such, administrative tasks are handled solely by Jenkins. As a result, our containers should be self-sufficient and tidy. That is, a container should leave behind nothing but logs and rely on nothing but environment variables provided by jenkins (ZUUL_URL, ZUUL_REF, etc).

As of November of 2017, we are working towards moving docker images from docker hub to docker-registry.wikimedia.org under the releng namespace.

Jenkins Agent Creation[edit]

  • Create a new instance in horizon with a name following the pattern 'integration-slave-docker-100X'
  • Wait for the first puppet run to complete and log in
  • Run the following to finish switching to the integration puppet master:
sudo rm -fR /var/lib/puppet/ssl
sudo mkdir -p /var/lib/puppet/client/ssl/certs
sudo puppet agent -tv
sudo cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/client/ssl/certs
sudo puppet agent -tv
  • Add the 'role::ci::slave::labs::docker' class to the instance in horizon
    • For larger instance types (m1.xlarge and bigram) specify true for the docker_lvm_volume parameter.
  • Run a final update for puppet 'sudo puppet agent -tv'
  • Pull an initial set of docker images onto the host (using latest tags) to avoid doing this in test runs:
sudo docker pull docker-registry.wikimedia.org/releng/castor:latest
sudo docker pull docker-registry.wikimedia.org/releng/quibble-stretch:latest
sudo docker pull docker-registry.wikimedia.org/wikimedia-stretch:latest
  • Add the slave in the jenkins UI

Image Creation[edit]

Images used in Wikimedia's CI system are created from the integration/config repository. As of Nov 15th 2017 we are in a transition period between using build.py in the integration/config repository, and using docker pkg.

Images using docker-pkg[edit]

docker-pkg is a python3 program that is used to build both CI images and production docker images via Jinja2 templating.

Installing docker-pkg, building docker-pkg images[edit]

  1. Clone the code from docker-pkg and install via pip3
    $ git clone https://gerrit.wikimedia.org/r/#/admin/projects/operations/docker-images/docker-pkg
    $ cd docker-pkg
    $ pip3 install -e .
    
  2. Clone the integration/config project
    $ git clone https://gerrit.wikimedia.org/r/integration/config
    
  3. Now you can build all images in the dockerfiles folder using docker-pkg
    $ cd path/to/integration/config
    $ docker-pkg -c dockerfiles/config.yaml dockerfiles
    

Example output:

== Step 0: scanning dockerfiles ==
Will build the following images:
* docker-registry.wikimedia.org/releng/ci-stretch:0.1.0
* docker-registry.wikimedia.org/releng/operations-puppet:0.1.0
* docker-registry.wikimedia.org/releng/ci-jessie:0.3.0
== Step 1: building images ==
=> Building image docker-registry.wikimedia.org/releng/ci-stretch:0.1.0
=> Building image docker-registry.wikimedia.org/releng/operations-puppet:0.1.0
=> Building image docker-registry.wikimedia.org/releng/ci-jessie:0.3.0
== Step 2: publishing ==
NOT publishing images as we have no auth setup
== Build done! ==
You can see the logs at ./docker-pkg-build.log

Publishing docker-pkg images[edit]

There is a new Fab task called deploy_docker. This task will build and publish all docker-pkg images in integation/config on contint1001

Testing new containers[edit]

Once the new container is pushed to docker hub it should be tested on one of the integration-slave-docker-100x machines. As of August 2017 there are 4 such machines: integration-slave-docker-100[1:4].

To test

  1. ssh to one of the integration-slave-docker machines and su to the jenkins-deploy user.
    you@laptop:~$ ssh integration-slave-docker-1004
    you@integration-slave-docker:~$ sudo su - jenkins-deploy
    
  2. Create a new directory and an environment file that contains the information passed from Jenkins in the form of ZUUL_* variables
    jenkins-deploy@integration-slave-docker:~$ mkdir docker-test && cd docker-test
    jenkins-deploy@integration-slave-docker:docker-test$ printf "ZUUL_PROJECT=operations/puppet\nZUUL_URL=git://contint2001.wikimedia.org\nZUUL_REF=refs/zuul/production/Ze59ae894f02248d9888835dbaa14dfdf\nZUUL_COMMIT=045fcb14e9fd7885957d900b9a97c883fc5cd26d\n" > .env
    
  3. Run the new docker container with the environment file and ensure that it runs correctly
    jenkins-deploy@integration-slave-docker:docker-test$ mkdir log
    jenkins-deploy@integration-slave-docker:docker-test$ docker run --rm -it --env-file .env --volume "$(pwd)"/log:/var/lib/jenkins/log contint/operations-puppet
    
  4. If everything is working as anticipated, update JJB with the Dockerfile version that has been pushed to the Wikimedia Docker registry.