-
André Anjos authoredAndré Anjos authored
Deploying a macOS-based CI for Bob
Warning
THIS INSTRUCTIONS ARE DEPRECATED. Using a VirtualBox for running the gitlab runner, at least with a base macOS as of 10.9 did not work reliably in our experience. Using Parallels Desktop may be an alternative, but at this point, it wasn't tried.
This document contains instructions to build and deploy a new virtual-machine-based CI for macOS. The instructions are divided into 2 parts: the creation of the base virtual-machine to use for the CI builds (the "guests") and how to deploy this newly created virtual-machine into a computer running macOS (the "host"). Instructions for deployment assume a freshly installed machine. There are no requirements on the OS for the host, for as long as it is supported by VirtualBox and gitlab runner.
Note
Data on this discussion is relevant to this setup. If you have problems, please check it.
Building the reference "guest" image
This package contains a script that can build provisioned VirtualBox images that contain a minimal installation of macOS for CI purposes. To build such images you'll need:
-
A computer running macOS (you can only virtualize macOS on another computer running macOS)
-
A recent version of Packer
-
The app for the OS you'll want to install. You can obtain installers for most macOS versions through the AppStore. Just install the app through the AppStore on the computer running this script.
-
The Xcode SDK you'll need installed on the image, downloaded to the
xcode
subdirectory of this package. For example, to download the latest release of macOS 10.9 (Mavericks) SDK, do this:$ mkdir xcode && cd xcode $ wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.13/MacOSX10.9.sdk.tar.xz
Once all is in place, run the build.sh
command, passing the version of
macOS you wish to build an image for:
$ ./build.sh 10.9
The build process is divided in 3 (cached) stages:
- Conversion of the macOS installation app into a bootable DMG
- Creation of a OVF file with the base macOS installation
- Creation of the final OVA file with the provisioned macOS installation. The
provisioning consists of the execution, in sequence, of all scripts in the
scripts
directory.
The virtual image, when ready, will be available at the ova
directory. The
virtual image contains the following elements:
- A "gitlab" user that is also part of the "staff" group (so it has
administrator priviledges on the machine, i.e., can sudo) is created. This
user has a
~/.ssh/authorized_keys
files pre-set so that it is possible to login passwordlessly to it. This is required by the Gitlab Runner VirtualBox Executor. - The Xcode Command-line Tools for the respective version of macOS is properly installed
- The 10.9 Xcode SDK kit is installed on
/opt/MacOSX10.9.sdk
. You must set your CI'sconda_build_config.yaml
accordingly. - homebrew is available on the image and accessible via the usual brew
command-line (
/usr/local/bin/brew
). The following packages are pre-installed: git, curl, python-twine. The "gitlab" user is the installer of homebrew. The value of/etc/paths
is fixed so that binaries in/usr/local/bin
are in priority - The
/etc/hosts
file is modified to take into account some internal Idiap settings - The system is updated to the latest version of the OS
- The Gitlab Runner is also installed into the virtual image so that artifact uploading works fine
- Automatic network detection is enabled
- Auto-login to the "gitlab" user is enabled
Building new guests
Currently, the build system can only handle macOS 10.9 guest systems. In order
to implement new types of guests, you'll need to inspect build.sh
and
increment it to support newer installation apps.
Deployment on the "host" system
These instructions assume you start your deployment from a newly installed macOS machine. They explain how to create a macOS-based gitlab runner CI machine deploying the virtual machine you created above.
Instructions
-
Disable all energy saving features. Go to "System Preferences" then "Energy Saver":
- Enable "Prevent computer from sleeping..."
- Disable "Put hard disks to sleep when possible"
- Leave "Wake for network access" enabled
- You may leave the display on sleep to 10 minutes
-
Create a new user (without administrative priviledges) called
gitlab
. Choose a password to protect access to this user -
Enable SSH access to the machine by going on
System Preferences
,Sharing
and then selectingRemote Login
. Make sure only users on theAdministrators
group can access the machine. -
Create as many
Administrator
users as required to manage the machine -
Copy the private SSH key from the virtual machine to
/Users/gitlab/.ssh/macos_vbox_rsa
. The availability of the private key on the host makes it able to login passwordlessly to the CI guests. Note this file is sensitive and should not be distributed publicly. If you don't know where this file is located, check one of the pre-installed machines by logging-in there and checking this file under thegitlab
user. If no machines are available, you'll need to create a new key-pair and reset the virtual image to allow logins with the newly generate public part of the pair. -
Install the Xcode Commandline Tools for macOS. Follow the script in
scripts/xcode-cli-tools.sh
for instructions on how to install that from the command-line. You'll need the command-line tools from macOS to install homebrew (see next point) -
Modify
/etc/paths
to move/usr/local/bin
up the stack -
Install homebrew (see
scripts/homebrew.sh
), thengit
,curl
andtwine-pypi
. Noticecurl
is a keg-only recipe and needs a special command-line to get it activated correctly (see the script for details). Once activated, logout and login again to getcurl
properly working. You may also want to optionally install the following homebrew packages:-
coreutils
(provide sane versions of unix commands, with theg
prefix - e.g.gls
instead ofls
) -
bash-completion
(provides auto-completion for most command commands) -
highlight
(for code-source highlighting on the command-line) -
neovim
(cleaner version of vim). Executebrew install python3 ruby neovim
. -
packer
(for building the VM as per above): Executebrew install packer packer-completion
-
-
Install VirtualBox. You don't need the extension pack, but it will complain a bit less if that is also installed. Once that is done, as
gitlab
user, import the output available in theova
directory in the previous section into VirtualBox. Annotate the virtual machine name for later (you'll use that during the gitlab runner configuration). You can do this from the command-line with the following command:$ VBoxManage import ova/macos109.ova $ VBoxManage list vms # to check the VM was imported
-
Install the gitlab runner. Make sure the service is deployed and started correctly. Make sure that the
gitlab
user is the one running the service by the end of the installation -
Install the minio service (this is required to enable build caching using the VirtualBox executor subsystem of the gitlab runner):
- Install homebrew services support by running
brew services
once - Install minio with homebrew (use tap
minio/stable/minio
). Don't use homebrew's default install - Start the minio service with
sudo brew services start minio
(yes, you needsudo
here) - Install minio-mc from the same homebrew tap (
minio/stable/mc
) - Retrieve access/secret from
sudo head /usr/local/etc/minio/config.json
- The next steps should be executed as the
gitlab
user, so login as that user physically on the machine or usesu - gitlab
:- Configure the access to the minio service with
mc config host add minio http://127.0.0.1:9000 <accessKey> <secretKey>
- Create a bucket to keep the cache produced by the VirtualBox
executors with
mc mb minio/caches
- Configure the access to the minio service with
- Install homebrew services support by running
-
Configure the runner for virtualbox executor, with caching based on minio. Follow the instructions for setting up the virtualbox executor for gitlab (as
gitlab
user, execute on the command-line:gitlab-runner stop
;vi .gitlab-runner/config.toml
;gitlab-runner start
). Once that is set, your runner configuration should look like this (remove comments if gitlab does not like them):concurrent = 8 # set this to the number of cores available check_interval = 10 # do **not** leave this to zero [[runners]] name = "<runner-name>" # use a suggestive name output_limit = 102400 # this value is in kb, so we mean 100 mb url = "https://gitlab.idiap.ch" # this is our gitlab service token = "abcdefabcdefabcdefabcdefabcdef" # this is specific to the conn. executor = "virtualbox" # select this builds_dir = "/Users/gitlab/builds" # set this or bugs occur cache_dir = "/Users/gitlab/caches" # this is optional, but keeps things organized [runners.ssh] # set this to the "private" key of the authorized one that is built-in # your virtual machine - you also need the user (otherwise root is # used!) user = "gitlab" identity_file = "/Users/gitlab/.ssh/macos_vbox_rsa" [runners.virtualbox] base_name = "macos109" # use the VM name from above disable_snapshots = true # to avoid accumulating build/cache data [runners.cache] # if you configured that as per above, leave this as is Type = "s3" ServerAddress = "10.0.2.2:9000" # use the host address from the internal VM network AccessKey = "<accessKey>" SecretKey = "<secretkey>" BucketName = "caches" Insecure = true Path = "minio" Shared = true
A Quick Note on Performance
The commands above are going to leave you with a virtual machine using a dynamically-allocated disk. This is not very efficient for its CI job. After importing the VM into VirtualBox, open the terminal and do the following to convert the dynamically allocated disk to a fixed-size disk:
$ VBoxManage list hdds
# Identify the location of the disk you want to change. Create a new disk
# with a similar name to the original and on the same base directory
$ VBoxManage clonehd <path-to-source>.vdmk <path-to-destination>.vdi –variant Fixed
# Detach the old disk from the image (by attaching "none")
$ VboxManage storageattach macos109 --storagectl "SATA Controller" --port 0 --medium none
# Delete the old disk (uuid can be obtained with "list hdds")
$ VBoxManage closemedium disk 87860667-35d9-4c1a-b447-e383b8464b96 --delete
# Attach the fixed size disk to the image
$ VBoxManage storageattach macos109 --storagectl "SATA Controller" --type hdd --port 0 --device 0 --medium <path-to-destination>.vdi
The disk performance should improve after this. If that is still not the case, check the Host I/O caching is enabled. This can also dramatically affect performance. Go to the virtual machine "Settings" and then "Storage" to check for this flag. If not set, you can modify the machine with this command:
$ VBoxManage storagectl macos109 --name "SATA Controller" --hostiocache on
Ensure that, when the snapshots for the machine are taken and the runner virtual machine copies are created, the flag remains ON.
Memory
By default, the virtual machine is set to take 3GB of RAM. It is not advised to max-out your machine's memory. Before launching the gitlab runner, ensure to set a proper limit on the base virtual machine. Leaving 4 to 6 GB of RAM for the OS is recommended.
Quick VBoxManage Command Reference
-
VBoxManage import <file>
- imports an OVA/OVF file -
VBoxManage list vms
- list all available VMs -
VBoxManage list runningvms
- lists all running VMs -
VBoxManage startvm <name> --type headless
- start VM in headless mode -
VBoxManage controlvm <name> poweroff
- power-off VM -
VBoxManage unregistervm <name> --delete
- remove VMs and associated disks - Add port forwarding:
-
machine is OFF:
VBoxManage modifyvm <name> --natpf1 "ssh,tcp,,2200,,22"
-
machine is ON:
VBoxManage controlvm <name> natpf1 "ssh,tcp,,2200,,22"
-
machine is OFF:
- Remove port forwarding:
-
machine is OFF:
VBoxManage modifyvm <name> --natpf1 delete ssh
-
machine is ON:
VBoxManage controlvm <name> natpf1 delete ssh
-
machine is OFF:
-
VBoxManage showvminfo <name> | grep ssh
- shows which port (if any) is open for SSH access when the machine is running
Updating the "guest"
You may update the guest base virtual machines by re-building a new OVA file
and re-importing it into the VirtualBox instance of the gitlab
user. Once
that is done, delete all other temporarily created images, stop the gitlab
runner and re-configure ~/.gitlab-runner/config.toml
to use the new
image. Re-start the runner with gitlab-runner start
to take those changes
into effect.
Note
For some unknown reason, gitlab-runner start
(stop, restart) must be
called physically on the machine. It does not work over an SSH connection.
Optionally, you may also add a new VirtualBox runner into your
configuration, in which case it is possible to test the new CI without
unconfiguring the previous setup. Just copy the [[runners]]
section above
and make sure to set a new unique name
and VirtualBox machine
base_name
. You may leave the [runenrs.cache]
copy untouched, which will
make both guests share the caching system. Otherwise, create a new bucket on
the server to separate caches.
Updating OpenSSH on the "guest"
It is possible to update OpenSSH on the guest, so that you get password-access to the image working properly. See my comment on this discussion for details. To do so, just edit scripts/homebrew.sh to uncomment the installation of homebrew's openssh and then add scripts/enable-openssh.sh to the provision step. At this point, the image will be ssh-able with a user and password setup instead of the user/identity_file that is advised above.