Continuous integration/Qemu

RunbookEdit

AgentEdit

To provision a new agent, follow Nova Resource:Integration/Setup to set up a Qemu worker agent for Jenkins.

The following step to create /srv/vm-images/sshkey_qemu_root_v1 is not yet puppetized:

agent$ ssh-keygen -t rsa -b 4096
  • Don't use any actual e-mail address (leave the default to your shell name and agent hostname).
  • Don't set a pass phrase.
  • Name it like "sshkey_qemu_<user>_v<sequence>", for example sshkey_qemu_root_v1.
  • Use sudo to move ~/.ssh/sshkey_qemu_root_v1 and ~/.ssh/sshkey_qemu_root_v1.pub to /srv/vm-images/.
  • Set chmod 644 *.img, so that it is available as read-only for the Jenkins user.

Create Qemu imageEdit

To boot into a Qemu-compatible virtual machine image for the first time:

# Install the needed tools on the outer VM host
apt-get install debootstrap qemu-system libguestfs-tools

# Create a minimal bullseye (or any other release) installation in ./vm-root
sudo debootstrap --include=linux-image-amd64,grub2,console-setup bullseye vm-root https://mirrors.wikimedia.org/debian/

# Edit the root filesystem as needed.  At a minimum, edit vm-root/etc/default/grub
# to enable serial support, and vm-root/etc/passwd to permit root login.

# Convert ./vm-root to a disk image, ./vm.img
sudo virt-make-fs --format qcow2 --type ext4 --partition --size +1G vm-root vm.img

# Now boot vm.img so that the MBR can be updated (replace kernel and initrd filenames
# with whatever was installed by debootstrap):
qemu-system-x86_64 -nographic -m 4G -hda vm.img -kernel vm-root/boot/vmlinuz-5.10.0-9-amd64 -initrd vm-root/boot/initrd.img-5.10.0-9-amd64 -append 'console=ttyS0 root=/dev/sda1'

Now, within the virtual machine image, run:

update-grub
grub-install /dev/sda

You can shut down the VM, or use C-a x to exit. The files in ./vm-root can now be deleted, and the image can be booted via:

qemu-system-x86_64 -nographic -m 4G -hda vm.img

The above instructions create an image 1GB larger than the base system. If you need to expand the image later, you can:

# Create a new disk, and resize the old one into the new one
truncate -s 20G ./out.img
virt-resize --expand /dev/sda1 ./vm.img ./out.img

# Verify the new disk size
virt-filesystems --long --parts --blkdevs -h -a ./out.img

Launch for remote controlEdit

This is how Jenkins jobs launch the VM. This is different from the above provisioning workflow. Rather than getting an interactive shell directly from Qemu, we we launch the Qemu with an SSH port exposed, and login that way:

# Terminal 1 (launch guest VM)
you@agent$ qemu-system-x86_64 -device virtio-net,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::4293-:22 -smp 4 -m 4096 -nographic vm.img
…

# Terminal 2 (create copy of key, then open shell via ssh)
you@agent$ install -m 600 /srv/vm-images/sshkey_qemu_root_v1 root.key 
you@agent$ ssh -i ./root.key -p 4293 root@localhost
…
root@img# …
root@img# exit

Create new snapshot versionEdit

Launch a snapshot manually to create changes to the base image. Never launch or modify a snapshot directly. Always copy first.

you@agent$ cp /path/to/thing-to-change.img ~/vm.img
you@agent$ qemu-system-x86_64 -m 4096 -nographic vm.img

Wait for the bootloader to pass, then once "img login:" appears, enter "root".

A shell will appear shortly. Make your changes, then run exit to log out from the VM.

Once back in the bootscreen, use Ctrl-A X to exit from Qemu and return to the agent.

The vm.img file is now an updated snaphot, ready for publishing.

Publish new snapshotEdit

  1. Before publishing a new snaphot, verify that you are able to launch the VM and connect to it over SSH using the "Launch for remote control" steps above.
  2. Use sudo to move the img file from your home directory to /srv/vm-images/qemu-<flavour>-<date><sequence>.img. For example, /srv/vm-images/qemu-debian10buster-2020_04_28a.img, or …-2020_04_28b.img, where the sequence letter is for revisions of the same base.
  3. Set chmod 644 *.img, so that it is available as read-only for the Jenkins user.

SnapshotsEdit

qemu-debian10busterEdit

Current version: qemu-debian10buster-2020_05_04c.img

Provisioning:

  • Snapshot of Debian 10 Buster with.
  • Grub configured to use a serial console.
  • Run apt-get update
  • Required packages: apt-get -y install ssh git docker.io
  • Optional packages:
    • Preload for mw-cli jobs (T248779):
      • apt-get -y install curl make python3-dev libffi-dev gcc libc-dev cargo
  • Ensure no containers are running:
    • docker ps
  • edit /etc/ssh/sshd_config, and set PermitRootLogin yes (The nano editor is pre-installed).
  • mkdir /root/.ssh
  • copy contents from host file sshkey_qemu_root_v1.pub to guest /root/.ssh/authorized_keys (E.g. create the file with nano and copy the pub file contents from the other terminal tab)
  • run systemctl restart sshd.service

Jenkins JobsEdit

The Jenkins jobs currently use https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/qemu-run.bash

After publishing a new image, update this script to point to the new image version.

When making changes to this script, you will need to regenerate the Jenkins jobs that embed this script for the changes to take effect. (see Continuous integration/Jenkins job builder).