Gitlab Docker CI Machine Installation

This guide assumes you want to install a new CI machine from scratch, with docker and docker build support, for Bob.

Steps

1. Install a base Debian 8.x distribution
2. Install the following packages:

   $ apt-get update
   $ apt-get install openssh-server curl vim-nox htop

3. Make sure to copy access keys for personel at /root/.ssh/authorized_keys
4. Access your machine via ssh from this point onwards, make sure it is OK
5. Follow the steps in section "Kernel Support" below for extended CPU and memory control on docker containers, in case you need those
6. If there are locale problems, install the missing locales with:

   $ dpkg-reconfigure locales

7. Install docker:

   $ curl -sSL https://get.docker.com/ | sh

8. Install Gitlab CI:

   $ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | bash

9. Add the gitlab-runner user to the docker group:

   $ usermod -aG docker gitlab-runner

10. Register the runner and fine-tune its configuration:

$ gitlab-ci-multi-runner register
$ vim /etc/gitlab-runner/config.toml
$ service gitlab-runner restart

If you'd like to user docker build, you'll need to register 2 runners: one with the docker executor and another with a shell executor (that can access docker build. More information here: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html 11. Configure docker to access Idiap's registry:

$ mkdir -p /etc/docker/certs.d/docker.idiap.ch
$ scp ./docker.crt root@<machine>:/etc/docker/certs.d/docker.idiap.ch/
$ scp ./gitlab.crt root@<machine>:/etc/docker/certs.d/docker.idiap.ch/

Kernel Support

If you need memory and CPU quota control over your Docker containers, you'll ned to adjust a few parameters on the running machine and, possibly, recompile the its kernel.

Here are some tests you can run to test if both are available:

$ for the memory check
$ docker run -it --rm -m 128M --memory-swap=128M progrium/stress --vm 1 --vm-bytes 130M --vm-hang 0
$ for CPU quota check
$ docker run -it --rm --cpu-quota=50000 --cpu-period=100000 progrium/stress --cpu 3

In each case, you should not see any warning message and the docker engine must respect the imposed limits (which you can verify with docker stats). If you see warning messages claiming a feature is not enabled, you must either enable the features either by recompiling the kernel or passing the right boot parameters to enable them. On a Debian system, you have to do the following:

Add cgroup_enable=memory swapaccount=1 as boot parameters. Reboot with those options. After rebooting, the first command-line above should start working. Reference: https://docs.docker.com/engine/installation/linux/ubuntulinux/#/enable-memory-and-swap-accounting

Recompile the stock kernel and add CPU bandwidth provisioning for FAIR_GROUP_SCHED under General setup -> Control group support -> Group CPU scheduler. Reboot with that modified kernel. After rebooting, the second command-line above should start working correctly. Reference: http://www.blaess.fr/christophe/2012/01/07/linux-3-2-cfs-cpu-bandwidth-english-version/

N.B.: Idiap Debian kernels have both enabled by default. In case you're running on a virtual machine with a single CPU, we have noticed the CFS can be a bit imprecise (e.g. allocate more than 50% CPU even if you set it to have only 50%). It works better if there are more CPUs assigned to the virtual machine in this case.

Recompilation on a stock Debian distribution:

$ apt-get install fakeroot kernel-package linux-source libncurses5-dev
$ cd /usr/src
$ tar xf linux-source-3.16.tar.xz
$ cd linux-source-3.16
$ xzcat ../linux-config-3.16/config.amd64_none_amd64.xz > .config
$ make menuconfig
# Do as explained above
# Save the new configuration
$ fakeroot make-kpkg --initrd --revision=1.0.bob kernel_image
$ dpkg -i ../linux-image-*.dpkg