Skip to content
Snippets Groups Projects
Select Git revision
  • master
1 result

README.rst

Blame
  • README.rst 15.17 KiB

    Deploying a macOS-based CI for Bob

    Warning

    THIS INSTRUCTIONS ARE DEPRECATED. Using a VirtualBox for running the gitlab runner, at least with a base macOS as of 10.9 did not work reliably in our experience. Using Parallels Desktop may be an alternative, but at this point, it wasn't tried.

    This document contains instructions to build and deploy a new virtual-machine-based CI for macOS. The instructions are divided into 2 parts: the creation of the base virtual-machine to use for the CI builds (the "guests") and how to deploy this newly created virtual-machine into a computer running macOS (the "host"). Instructions for deployment assume a freshly installed machine. There are no requirements on the OS for the host, for as long as it is supported by VirtualBox and gitlab runner.

    Note

    Data on this discussion is relevant to this setup. If you have problems, please check it.

    Building the reference "guest" image

    This package contains a script that can build provisioned VirtualBox images that contain a minimal installation of macOS for CI purposes. To build such images you'll need:

    • A computer running macOS (you can only virtualize macOS on another computer running macOS)

    • A recent version of Packer

    • The app for the OS you'll want to install. You can obtain installers for most macOS versions through the AppStore. Just install the app through the AppStore on the computer running this script.

    • The Xcode SDK you'll need installed on the image, downloaded to the xcode subdirectory of this package. For example, to download the latest release of macOS 10.9 (Mavericks) SDK, do this:

      $ mkdir xcode && cd xcode
      $ wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.13/MacOSX10.9.sdk.tar.xz

    Once all is in place, run the build.sh command, passing the version of macOS you wish to build an image for:

    $ ./build.sh 10.9

    The build process is divided in 3 (cached) stages:

    • Conversion of the macOS installation app into a bootable DMG
    • Creation of a OVF file with the base macOS installation
    • Creation of the final OVA file with the provisioned macOS installation. The provisioning consists of the execution, in sequence, of all scripts in the scripts directory.

    The virtual image, when ready, will be available at the ova directory. The virtual image contains the following elements:

    1. A "gitlab" user that is also part of the "staff" group (so it has administrator priviledges on the machine, i.e., can sudo) is created. This user has a ~/.ssh/authorized_keys files pre-set so that it is possible to login passwordlessly to it. This is required by the Gitlab Runner VirtualBox Executor.
    2. The Xcode Command-line Tools for the respective version of macOS is properly installed
    3. The 10.9 Xcode SDK kit is installed on /opt/MacOSX10.9.sdk. You must set your CI's conda_build_config.yaml accordingly.
    4. homebrew is available on the image and accessible via the usual brew command-line (/usr/local/bin/brew). The following packages are pre-installed: git, curl, python-twine. The "gitlab" user is the installer of homebrew. The value of /etc/paths is fixed so that binaries in /usr/local/bin are in priority
    5. The /etc/hosts file is modified to take into account some internal Idiap settings
    6. The system is updated to the latest version of the OS
    7. The Gitlab Runner is also installed into the virtual image so that artifact uploading works fine
    8. Automatic network detection is enabled
    9. Auto-login to the "gitlab" user is enabled

    Building new guests

    Currently, the build system can only handle macOS 10.9 guest systems. In order to implement new types of guests, you'll need to inspect build.sh and increment it to support newer installation apps.

    Deployment on the "host" system

    These instructions assume you start your deployment from a newly installed macOS machine. They explain how to create a macOS-based gitlab runner CI machine deploying the virtual machine you created above.

    Instructions

    1. Disable all energy saving features. Go to "System Preferences" then "Energy Saver":

      • Enable "Prevent computer from sleeping..."
      • Disable "Put hard disks to sleep when possible"
      • Leave "Wake for network access" enabled
      • You may leave the display on sleep to 10 minutes
    2. Create a new user (without administrative priviledges) called gitlab. Choose a password to protect access to this user

    3. Enable SSH access to the machine by going on System Preferences, Sharing and then selecting Remote Login. Make sure only users on the Administrators group can access the machine.

    4. Create as many Administrator users as required to manage the machine

    5. Copy the private SSH key from the virtual machine to /Users/gitlab/.ssh/macos_vbox_rsa. The availability of the private key on the host makes it able to login passwordlessly to the CI guests. Note this file is sensitive and should not be distributed publicly. If you don't know where this file is located, check one of the pre-installed machines by logging-in there and checking this file under the gitlab user. If no machines are available, you'll need to create a new key-pair and reset the virtual image to allow logins with the newly generate public part of the pair.

    6. Install the Xcode Commandline Tools for macOS. Follow the script in scripts/xcode-cli-tools.sh for instructions on how to install that from the command-line. You'll need the command-line tools from macOS to install homebrew (see next point)

    7. Modify /etc/paths to move /usr/local/bin up the stack

    8. Install homebrew (see scripts/homebrew.sh), then git, curl and twine-pypi. Notice curl is a keg-only recipe and needs a special command-line to get it activated correctly (see the script for details). Once activated, logout and login again to get curl properly working. You may also want to optionally install the following homebrew packages:

      • coreutils (provide sane versions of unix commands, with the g prefix - e.g. gls instead of ls)
      • bash-completion (provides auto-completion for most command commands)
      • highlight (for code-source highlighting on the command-line)
      • neovim (cleaner version of vim). Execute brew install python3 ruby neovim.
      • packer (for building the VM as per above): Execute brew install packer packer-completion
    9. Install VirtualBox. You don't need the extension pack, but it will complain a bit less if that is also installed. Once that is done, as gitlab user, import the output available in the ova directory in the previous section into VirtualBox. Annotate the virtual machine name for later (you'll use that during the gitlab runner configuration). You can do this from the command-line with the following command:

      $ VBoxManage import ova/macos109.ova
      $ VBoxManage list vms # to check the VM was imported
    10. Install the gitlab runner. Make sure the service is deployed and started correctly. Make sure that the gitlab user is the one running the service by the end of the installation

    11. Install the minio service (this is required to enable build caching using the VirtualBox executor subsystem of the gitlab runner):

      1. Install homebrew services support by running brew services once
      2. Install minio with homebrew (use tap minio/stable/minio). Don't use homebrew's default install
      3. Start the minio service with sudo brew services start minio (yes, you need sudo here)
      4. Install minio-mc from the same homebrew tap (minio/stable/mc)
      5. Retrieve access/secret from sudo head /usr/local/etc/minio/config.json
      6. The next steps should be executed as the gitlab user, so login as that user physically on the machine or use su - gitlab:
        1. Configure the access to the minio service with mc config host add minio http://127.0.0.1:9000 <accessKey> <secretKey>
        2. Create a bucket to keep the cache produced by the VirtualBox executors with mc mb minio/caches
    12. Configure the runner for virtualbox executor, with caching based on minio. Follow the instructions for setting up the virtualbox executor for gitlab (as gitlab user, execute on the command-line: gitlab-runner stop; vi .gitlab-runner/config.toml; gitlab-runner start). Once that is set, your runner configuration should look like this (remove comments if gitlab does not like them):

      concurrent = 8  # set this to the number of cores available
      check_interval = 10  # do **not** leave this to zero
      
      [[runners]]
        name = "<runner-name>"  # use a suggestive name
        output_limit = 102400  # this value is in kb, so we mean 100 mb
        url = "https://gitlab.idiap.ch"  # this is our gitlab service
        token = "abcdefabcdefabcdefabcdefabcdef"  # this is specific to the conn.
        executor = "virtualbox"  # select this
        builds_dir = "/Users/gitlab/builds"  # set this or bugs occur
        cache_dir = "/Users/gitlab/caches"  # this is optional, but keeps things organized
        [runners.ssh]
          # set this to the "private" key of the authorized one that is built-in
          # your virtual machine - you also need the user (otherwise root is
          # used!)
          user = "gitlab"
          identity_file = "/Users/gitlab/.ssh/macos_vbox_rsa"
        [runners.virtualbox]
          base_name = "macos109"  # use the VM name from above
          disable_snapshots = true  # to avoid accumulating build/cache data
        [runners.cache]  # if you configured that as per above, leave this as is
          Type = "s3"
          ServerAddress = "10.0.2.2:9000"  # use the host address from the internal VM network
          AccessKey = "<accessKey>"
          SecretKey = "<secretkey>"
          BucketName = "caches"
          Insecure = true
          Path = "minio"
          Shared = true

    A Quick Note on Performance

    The commands above are going to leave you with a virtual machine using a dynamically-allocated disk. This is not very efficient for its CI job. After importing the VM into VirtualBox, open the terminal and do the following to convert the dynamically allocated disk to a fixed-size disk:

    $ VBoxManage list hdds
    # Identify the location of the disk you want to change. Create a new disk
    # with a similar name to the original and on the same base directory
    $ VBoxManage clonehd <path-to-source>.vdmk <path-to-destination>.vdi –variant Fixed
    # Detach the old disk from the image (by attaching "none")
    $ VboxManage storageattach macos109 --storagectl "SATA Controller" --port 0 --medium none
    # Delete the old disk (uuid can be obtained with "list hdds")
    $ VBoxManage closemedium disk 87860667-35d9-4c1a-b447-e383b8464b96 --delete
    # Attach the fixed size disk to the image
    $ VBoxManage storageattach macos109 --storagectl "SATA Controller" --type hdd --port 0 --device 0 --medium <path-to-destination>.vdi

    The disk performance should improve after this. If that is still not the case, check the Host I/O caching is enabled. This can also dramatically affect performance. Go to the virtual machine "Settings" and then "Storage" to check for this flag. If not set, you can modify the machine with this command:

    $ VBoxManage storagectl macos109 --name "SATA Controller" --hostiocache on

    Ensure that, when the snapshots for the machine are taken and the runner virtual machine copies are created, the flag remains ON.

    Memory

    By default, the virtual machine is set to take 3GB of RAM. It is not advised to max-out your machine's memory. Before launching the gitlab runner, ensure to set a proper limit on the base virtual machine. Leaving 4 to 6 GB of RAM for the OS is recommended.

    Quick VBoxManage Command Reference

    • VBoxManage import <file> - imports an OVA/OVF file
    • VBoxManage list vms - list all available VMs
    • VBoxManage list runningvms - lists all running VMs
    • VBoxManage startvm <name> --type headless - start VM in headless mode
    • VBoxManage controlvm <name> poweroff - power-off VM
    • VBoxManage unregistervm <name> --delete - remove VMs and associated disks
    • Add port forwarding:
      • machine is OFF: VBoxManage modifyvm <name> --natpf1 "ssh,tcp,,2200,,22"
      • machine is ON: VBoxManage controlvm <name> natpf1 "ssh,tcp,,2200,,22"
    • Remove port forwarding:
      • machine is OFF: VBoxManage modifyvm <name> --natpf1 delete ssh
      • machine is ON: VBoxManage controlvm <name> natpf1 delete ssh
    • VBoxManage showvminfo <name> | grep ssh - shows which port (if any) is open for SSH access when the machine is running

    Updating the "guest"

    You may update the guest base virtual machines by re-building a new OVA file and re-importing it into the VirtualBox instance of the gitlab user. Once that is done, delete all other temporarily created images, stop the gitlab runner and re-configure ~/.gitlab-runner/config.toml to use the new image. Re-start the runner with gitlab-runner start to take those changes into effect.

    Note

    For some unknown reason, gitlab-runner start (stop, restart) must be called physically on the machine. It does not work over an SSH connection.

    Optionally, you may also add a new VirtualBox runner into your configuration, in which case it is possible to test the new CI without unconfiguring the previous setup. Just copy the [[runners]] section above and make sure to set a new unique name and VirtualBox machine base_name. You may leave the [runenrs.cache] copy untouched, which will make both guests share the caching system. Otherwise, create a new bucket on the server to separate caches.

    Updating OpenSSH on the "guest"

    It is possible to update OpenSSH on the guest, so that you get password-access to the image working properly. See my comment on this discussion for details. To do so, just edit scripts/homebrew.sh to uncomment the installation of homebrew's openssh and then add scripts/enable-openssh.sh to the provision step. At this point, the image will be ssh-able with a user and password setup instead of the user/identity_file that is advised above.