[doc] Move CI installation info to documentation

MANIFEST.in

 include LICENSE README.rst buildout.cfg version.txt
-recursive-include doc conf.py *.rst
+recursive-include doc conf.py *.rst *.sh
 recursive-include bob/devtools/data *.md *.yaml *condarc *.pem matplotlibrc

doc/ci.rst

+.. vim: set fileencoding=utf-8 :
+
+.. _bob.devtools.ci:
+
+
+==========================================
+ Instructions for CI machine installation
+==========================================
+
+This document contains instructions for CI machine installation from sources.
+
+
+.. toctree::
+   :maxdepth: 2
+
+   linux
+   macos

doc/index.rst

@@ -17,6 +17,7 @@ Documentation
 
    release
    api
+   ci
 
 
 Indices and tables

doc/links.rst

@@ -3,3 +3,6 @@
 .. Place here references to all citations in lower case
 
 .. _bob: https://www.idiap.ch/software/bob
+.. _shell executor: https://docs.gitlab.com/runner/executors/shell.html
+.. _gitlab runner: https://docs.gitlab.com/runner/install/osx.html
+.. _docker for mac: https://docs.docker.com/docker-for-mac/install/

doc/linux.rst

+.. vim: set fileencoding=utf-8 :
+
+.. _bob.devtools.ci.linux:
+
+============================
+ Deploying a Linux-based CI
+============================
+
+This document contains instructions to build and deploy a new bare-OS CI for
+Linux.  Instructions for deployment assume a freshly installed machine, with
+Debian 9.x running.  Our builds use Docker images.  We also configure
+docker-in-docker to enable to run docker builds (and other tests) within docker
+images.
+
+
+Docker and Gitlab-runner setup
+------------------------------
+
+Just follow the advices from https://medium.com/@tonywooster/docker-in-docker-in-gitlab-runners-220caeb708ca
+
+
+Hosts section
+=============
+
+We re-direct calls to www.idiap.ch to our internal server, for speed.  Just add
+this to `/etc/hosts`:
+
+.. code-block:: sh
+
+   $ echo "" >> /etc/hosts
+   $ echo "#We fake www.idiap.ch to keep things internal" >> /etc/hosts
+   $ echo "172.31.100.235 www.idiap.ch" >> /etc/hosts
+   $ echo "2001:620:7a3:600:0:acff:fe1f:64eb www.idiap.ch" >> /etc/hosts
+
+
+Gitlab runner configuration
+===========================
+
+We are currently using this:
+
+.. code-block:: ini
+
+   concurrent = 4
+   check_interval = 10
+
+   [[runners]]
+     name = "docker"
+     output_limit = 102400
+     url = "https://gitlab.idiap.ch/ci"
+     token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+     executor = "docker"
+     limit = 4
+     builds_dir = "/local/builds"
+     cache_dir = "/local/cache"
+     [runners.docker]
+       tls_verify = false
+       image = "continuumio/conda-concourse-ci"
+       privileged = false
+       disable_cache = false
+       volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/local/cache"]
+       extra_hosts = ["www.idiap.ch:172.31.100.235"]
+     [runners.cache]
+        Insecure = false
+
+   [[runners]]
+     name = "docker-build"
+     output_limit = 102400
+     executor = "shell"
+     shell = "bash"
+     url = "https://gitlab.idiap.ch/ci"
+     token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+     limit = 4
+     builds_dir = "/local/builds"
+     cache_dir = "/local/cache"
+
+
+Crontabs
+========
+
+.. code-block:: sh
+
+   # crontab -l
+   MAILTO=""
+   @reboot /root/docker-cleanup-service.sh
+   0 0 * * * /root/docker-cleanup.sh
+
+
+The `docker-cleanup-service.sh` is:
+
+.. code-block:: sh
+
+   #!/usr/bin/env sh
+
+   # Continuously running image to ensure minimal space is available
+
+   docker run -d \
+       -e LOW_FREE_SPACE=30G \
+       -e EXPECTED_FREE_SPACE=50G \
+       -e LOW_FREE_FILES_COUNT=2097152 \
+       -e EXPECTED_FREE_FILES_COUNT=4194304 \
+       -e DEFAULT_TTL=60m \
+       -e USE_DF=1 \
+       --restart always \
+       -v /var/run/docker.sock:/var/run/docker.sock \
+       --name=gitlab-runner-docker-cleanup \
+       quay.io/gitlab/gitlab-runner-docker-cleanup
+
+The `docker-cleanup.sh` is:
+
+.. code-block:: sh
+
+   #!/usr/bin/env sh
+
+   # Cleans-up docker stuff which is not being used
+
+   # Exited machines which are still dangling
+   #Caches are containers that we do not want to delete here
+   #echo "Cleaning exited machines..."
+   #docker rm -v $(docker ps -a -q -f status=exited)
+
+   # Unused image leafs
+   echo "Removing unused image leafs..."
+   docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

doc/macos-ci-install/admin-install.sh

+#!/usr/bin/env bash
+
+# Installs basic software on a fresh macOS installation that requires admin
+# priviledges.
+
+# VARIABLES - edit to your requirements
+MACOS_VERSION="${1}"
+USERNAME="${2}"
+
+# --------------------
+# Don't edit past this
+# --------------------
+
+# gets the current path leading to this script
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+
+echo "Setting up date and time..."
+${DIR}/datetime.sh
+
+echo "Updating the current system and disabling automatic updates..."
+${DIR}/system-update.sh
+
+echo "Installing Xcode Command-Line (CLI) tools..."
+${DIR}/xcode-cli-tools.sh
+
+echo "Installing macOS ${MACOS_VERSION} SDK..."
+${DIR}/install-sdk.sh ${MACOS_VERSION}
+
+echo "Setting up special idiap.ch host..."
+${DIR}/idiap-host.sh
+
+echo "Installing homebrew and build-time dependencies...":w
+${DIR}/setup-paths.sh
+${DIR}/install-homebrew.sh ${USER}
+
+echo "Installing (or updating) gitlab runner..."
+{DIR}/install-gitlab-runner.sh

doc/macos-ci-install/datetime.sh

+#!/usr/bin/env bash
+
+#    run more easily.
+# -e will cause the script to exit immediately if any command within it exits non 0
+# -o pipefail : this will cause the script to exit with the last exit code run.
+#               In tandem with -e, it will return the exit code of the first
+#               failing command.
+set -eox pipefail
+
+systemsetup -setusingnetworktime on
+systemsetup -settimezone Europe/Zurich
+systemsetup -setnetworktimeserver time.euro.apple.com.
+

doc/macos-ci-install/idiap-host.sh

+#!/usr/bin/env bash
+
+if [[ `grep -c "www.idiap.ch" /etc/hosts` != 0 ]]; then
+  echo "Not updating /etc/hosts - www.idiap.ch is already present..."
+else
+  echo "Updating /etc/hosts..."
+  echo "" >> /etc/hosts
+  echo "#We fake www.idiap.ch to keep things internal" >> /etc/hosts
+  echo "172.31.100.235 www.idiap.ch" >> /etc/hosts
+  echo "2001:620:7a3:600:0:acff:fe1f:64eb www.idiap.ch" >> /etc/hosts
+fi

doc/macos-ci-install/install-gitlab-runner.sh

+#!/usr/bin/env bash
+set -eox pipefail
+LOCATION=/usr/local/bin/gitlab-runner
+curl --output ${LOCATION} https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64
+chmod +x ${LOCATION}

doc/macos-ci-install/install-homebrew.sh

+#!/usr/bin/env bash
+
+set -x
+
+if [[ $EUID == 0 ]]; then
+  # changes path setup for all users, puts homebrew first
+  sed -e '/^\/usr\/local/d' -i .orig /etc/paths
+  echo -e "/usr/local/bin\n/usr/local/sbin\n/usr/local/opt/coreutils/libexec/gnubin\n$(cat /etc/paths)" > /etc/paths
+
+  # restarts to install brew as non-root user
+  exec su ${1} -c "$(which bash) ${0}"
+fi
+
+ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" </dev/null
+brew=/usr/local/bin/brew
+
+${brew} install curl git coreutils bash-completion highlight neovim tmux htop python3
+${brew} link --force curl #keg-only recipe
+${brew} cask install mactex

doc/macos-ci-install/install-sdk.sh

+#!/usr/bin/env bash
+
+# Installs the relevant SDK
+
+# gets the current path leading to this script
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+
+if [ ! -r "MacOSX${1}.sdk.tar.xz" ]; then
+  echo "Downloading macOS ${1} SDK..."
+  curl -L -o ${DIR}/MacOSX${1}.sdk.tar.xz https://github.com/phracker/MacOSX-SDKs/releases/download/10.13/MacOSX${1}.sdk.tar.xz
+else
+  echo "File MacOSX${1}.sdk.tar.xz is already here, skip download"
+fi
+
+if [ ! -d /opt ]; then mkdir /opt; fi
+cd /opt
+tar xfJ "${DIR}/MacOSX${1}.sdk.tar.xz"
+ln -s /opt/MacOSX${1}.sdk /Library/Developer/CommandLineTools/SDKs/

doc/macos-ci-install/setup-paths.sh

+#!/usr/bin/env bash
+
+set -x
+
+# changes path setup for all users, puts homebrew first
+sed -e '/^\/usr\/local/d' -i .orig /etc/paths
+sed -e '/^\/Library\/TeX\/texbin/d' -i .orig /etc/paths
+echo -e "/usr/local/bin\n/usr/local/sbin\n/usr/local/opt/coreutils/libexec/gnubin\n/Library/TeX/texbin\n$(cat /etc/paths)" > /etc/paths

doc/macos-ci-install/system-update.sh

+#!/bin/bash
+
+set -eox pipefail
+
+# read into array ${RECOMMENDED[@]}, a filtered list of available and
+# recommended software updates
+self=`basename $0`
+TMPFILE=`mktemp -t ${self}` || exit 1
+softwareupdate -l | grep -e '^\s\+\*' | sed -e 's/^[ \*]*//g' > ${TMPFILE}
+
+# starts of strings of installables we don't care about
+BLACKLIST=()
+BLACKLIST+=('iBook')
+BLACKLIST+=('iTunes')
+BLACKLIST+=('Install macOS')
+
+cat ${TMPFILE} | while read k; do
+  update="yes"
+  for l in "${BLACKLIST[@]}"; do
+    if [[ "${k}" == ${l}* ]]; then
+      update="no"
+    fi
+  done
+  if [[ "${update}" == "yes" ]]; then
+    echo "Updating $k..."
+    touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
+    softwareupdate --verbose --install "${k}"
+    rm /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
+  else
+    echo "Ignoring update for $k..."
+    softwareupdate --ignore "${k}"
+  fi
+done
+rm -f ${TMPFILE}
+
+# We don't want our system changing on us or restarting to update. Disable
+# automatic updates.
+echo "Disable automatic updates…"
+softwareupdate --schedule off

doc/macos-ci-install/xcode-cli-tools.sh

+#!/bin/sh
+
+set -eox pipefail
+
+# create the placeholder file that's checked by CLI updates' .dist code
+# in Apple's SUS catalog
+touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
+
+# find the CLI Tools update
+PROD=$(softwareupdate -l | grep "\*.*Command Line" | tail -n 1 | awk -F"*" '{print $2}' | sed -e 's/^ *//' | tr -d '\n')
+# install it
+softwareupdate -i "$PROD" --verbose
+
+rm /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress

doc/macos.rst

+.. vim: set fileencoding=utf-8 :
+
+.. _bob.devtools.ci.macos:
+
+
+============================
+ Deploying a macOS-based CI
+============================
+
+This document contains instructions to build and deploy a new bare-OS CI for
+macOS.  Instructions for deployment assume a freshly installed machine.
+
+
+.. note::
+
+   For sanity, don't use an OS with lower version number than the macOS SDK
+   code that will be installed (currently 10.9).  There may be undesired
+   consequences.  You may use the latest OS version in case of doubt, but by
+   default we recommend the one before the last stable version, for stability.
+   So, if the current version is 10.14, a good base install would use 10.13.
+
+
+Building the reference setup
+----------------------------
+
+0. Make sure the computer name is correctly set or execute the following on the
+   command-line, as an admin user::
+
+     $ sudo scutil --get LocalHostName
+     ...
+     $ sudo scutil --get HostName
+     ...
+     $ sudo scutil --get ComputerName
+     ...
+
+     # if applicable, run the following commands
+
+     $ sudo scutil --set LocalHostName "<hostname-without-domain-name>"
+     $ sudo scutil --set HostName "<fully-qualified-domain-name>"
+     $ sudo scutil --set ComputerName "<fully-qualified-domain-name>"
+
+1. Disable all energy saving features. Go to "System Preferences" then "Energy
+   Saver":
+
+   - Enable "Prevent computer from sleeping..."
+   - Disable "Put hard disks to sleep when possible"
+   - Leave "Wake for network access" enabled
+   - You may leave the display on sleep to 10 minutes
+2. Create a new user (without administrative priviledges) called ``gitlab``.
+   Choose a password to protect access to this user.  In "Login Options",
+   select this user to auto-login, type its password to confirm
+3. Enable SSH access to the machine by going on ``System Preferences``,
+   ``Sharing`` and then selecting ``Remote Login``. Make sure only users on the
+   ``Administrators`` group can access the machine.
+4. Create as many ``Administrator`` users as required to manage the machine
+5. Login as administrator of the machine (so, not on the `gitlab` account).  As
+   that user, run the ``admin-install.sh`` script (after copying this repo from
+   https://gitlab.idiap.ch/bob/bob.devtools via a zip file download)::
+
+     $ cd
+     $ unzip ~/Downloads/bob.devtools-master.zip
+     $ cd bob.devtools-master/doc/macos-ci-install
+     $ sudo ./admin-install.sh 10.9 gitlab
+
+   Check that script for details on what is installed and the order.  You may
+   execute pieces of the script by hand if something fails.  In that case,
+   please investigate why it fails and properly fix the scripts so the next
+   install runs more smoothly.
+6. Enter as gitlab user and install/configure the `gitlab runner`_:
+
+   Configure the runner for `shell executor`_, with local caching.  As
+   ``gitlab`` user, execute on the command-line::
+
+     $ gitlab-runner stop
+     $ vi .gitlab-runner/config.toml
+     $ gitlab-runner start
+
+   Once that is set, your runner configuration should look like this (remove
+   comments if gitlab does not like them)::
+
+      concurrent = 8  # set this to the number of cores available
+      check_interval = 10  # do **not** leave this to zero
+
+      [[runners]]
+        name = "<runner-name>"  # use a suggestive name
+        output_limit = 102400  # this value is in kb, so we mean 100 mb
+        url = "https://gitlab.idiap.ch"  # this is our gitlab service
+        token = "abcdefabcdefabcdefabcdefabcdef"  # this is specific to the conn.
+        executor = "shell"  # select this
+        builds_dir = "/Users/gitlab/builds"  # set this or bugs occur
+        cache_dir = "/Users/gitlab/caches"  # this is optional, but desirable
+        shell = "bash"
+7. While at the gitlab user, install `Docker for Mac`_.  Ensure to set it up to
+   start at login.  In "Preferences > Filesystem Sharing", ensure that
+   `/var/folders` is included in the list (that is the default location for
+   temporary files in macOS).
+8. Reboot the machine. At this point, the gitlab user should be auto-logged and
+   the runner process should be executing.  Congratulations, you're done!
+
+
+.. include:: links.rst