Continuous integration/Qemu

Agent

See Nova Resource:Integration/Setup for how to set up a Qemu worker node for Jenkins.

Snapshot maintenance

Provisioning

Launch a snapshot manually to create changes to the base image. Never launch or modify a snapshot directly. Always copy first.

you@agent$ cp /path/to/thing-to-change.img ~/vm.img
you@agent$ qemu-system-x86_64 -m 4096 -nographic vm.img

Wait for the bootloader to pass, then once "img login:" appears, enter "root".

A shell will appear shortly. Make your changes, then run exit to log out from the VM.

Once back in the bootscreen, use Ctrl-A X to exit from Qemu and return to the agent.

The vm.img file is now an updated snaphot, ready for publishing.

Launch for remote control

This is how Jenkins jobs launch the VM. This is different from the above provisioning workflow. Rather than getting an interactive shell directly from Qemu, we we launch the Qemu with an SSH port exposed, and login that way:

# Terminal 1
you@agent$ qemu-system-x86_64 -device virtio-net,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::4293-:22 -smp 4 -m 4096 -nographic vm.img
…

# Terminal 2
you@agent$ install -m 600 /srv/vm-images/sshkey_qemu_root_v1 root.key 
you@agent$ ssh -i ./root.key -p 4293 root@localhost
…
root@img# …
root@img# exit

Publish new snapshot

1. Before publishing a new snaphot, verify that you are able to launch the VM and connect to it over SSH using the "Launch for remote control" steps above.
2. Use sudo to move the img file from your home directory to /srv/vm-images/qemu-<flavour>-<date><sequence>.img. For example, /srv/vm-images/qemu-debian10buster-2020_04_28a.img, or …-2020_04_28b.img, where the sequence letter is for revisions of the same base.
3. Set chmod 644 *.img, so that it is available as read-only for the Jenkins user.

SSH Keys

To create a new keypair:

agent$ ssh-keygen -t rsa -b 4096

• Don't use any actual e-mail address (leave the default to your shell name and agent hostname).
• Don't set a pass phrase.
• Name it like "sshkey_qemu_<user>_v<sequence>", for example sshkey_qemu_root_v1.
• Use sudo to move ~/.ssh/sshkey_qemu_root_v1 and ~/.ssh/sshkey_qemu_root_v1.pub to /srv/vm-images/.
• Set chmod 644 *.img, so that it is available as read-only for the Jenkins user.

Snapshots

qemu-debian10buster

Current version: qemu-debian10buster-2020_05_04c.img

Provisioning:

• Snapshot of Debian 10 Buster with.
• Grub configured to use a serial console.
• Required packages:
  • apt-get -y install ssh git docker.io
• Optional
  • apt-get update
  • Preload packages to optimize mw-cli jobs (T248779):
    • apt-get -y install curl make python3-dev libffi-dev gcc libc-dev cargo
  • Ensure no containers are running:
    • docker ps
  • Prune any cached docker layers from your experimentation or from previous provisioning:
    • docker rmi $(docker images -q)
  • Preload docker parent layers used by node-fresh:
    • docker pull docker-registry.wikimedia.org/releng/node10-test-browser:0.6.3-s2
  • Preload docker parent layers used by mw-cli jobs:
    • docker pull docker-registry.wikimedia.org/dev/stretch-php72-fpm:2.0.0
    • docker pull docker-registry.wikimedia.org/dev/stretch-apache2:1.0.0-s1
    • docker pull docker-registry.wikimedia.org/dev/stretch-php72-jobrunner:2.0.0
• edit /etc/ssh/sshd_config, and set PermitRootLogin yes (The nano editor is pre-installed).
• mkdir /root/.ssh
• copy sshkey_qemu_root_v1.pub to /root/.ssh/authorized_keys (E.g. create the file with nano and copy the pub file contents from another tab)
• run systemctl restart sshd.service

Resizing the .img (outside of the image VM):

# Install the needed tool on the VM
apt-get install libguestfs-tools

# Create a new disk, and resize the old one into the new one
truncate -s 20G ./out.img
virt-resize --expand /dev/sda1 ./vm.img ./out.img

# Verify the new disk size
virt-filesystems --long --parts --blkdevs -h -a ./out.img

Jenkins Jobs

All Qemu jobs currently use https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/qemu-run.bash

If this file is updated, you will need to update all jenkins jobs that use the file so that changes take effect.