Conda-package-based CI

Ideally we want to replace our wheels system with conda packages and maintain conda recipes in the project folder itself.

The procedure will be:

• Each package will have a folder called conda and its recipe will be there.
• The CI will pickup that recipe and test the package based on that.
• Documentation is also built during testing the conda package.
• The built conda package will be used for testing other packages (replacing wheels) and it will also be deployed in our conda channel.
  • Make sure extra files (like db.sql3) are included in the conda package.
• Deploy the PyPI source, docs, and the conda package.

The logistics of this are fairly figured out and an initial implementation is available in bob.buildout!24 (merged).

To make this process easier, achieve two goals at the same time, and avoid hacks we need to switch from conda-build 2 to conda-build 3.

• This switch also implies that we need to switch from old C++ ABI to the new one.
• Also implies that we will use GCC 7 on Linux and Clang 5 on Mac which is cutting edge.

Another thing that we need to care about is keeping the new builds separate from the old builds (at least for now) since their ABI differences could break installations and our CI.

• Provide a stable and beta channel for our new packages.

Before we move into switching bob to the new ABI and compilers, we need to compile our dependencies first. I have checked and most of our dependencies (except for caffe) from the defaults are already built with the new ABI. So we are left with compiling the dependencies that we maintain in bob.conda. This is major part that I think is blocking our transition. Here is a list of packages that need to be recompiled:

• cyvlfeat
• dlib
• ffmpeg
• giflib
• ipdb
• kaldi
• keras
• keras-gpu
• libblitz
• libmatio
• libogg
• libsvm
• madmom
• menpo
• menpofit
• menpowidgets
• mne
• openfst
• pathlib
• pkgtools
• pyedflib
• schema
• sox
• speex
• tqdm
• vlfeat
• x264
• yasm
• zc.buildout
• zc.recipe.egg

Some of them of course being pure Python should be a breeze to convert.

To start the conversion of our dependencies (after the new conda channel is ready), people need to take one the dependencies listed above and convert them to conda build 3 and using the new compilers. We will do this in a branch called conda-build-3. Here are the instructions:

• Create a new branch for one of the dependencies (branching off form conda-build-3 and not master).
• Update the recipe to conda build 3 and make sure it compiles with the new compilers.
• How to build packages is here: https://conda.io/docs/user-guide/tasks/build-packages/index.html
• Remove the toolchain package from the list of dependencies and make sure you are using the new compiler packages: https://conda.io/docs/user-guide/tasks/build-packages/variants.html#compiler-packages
• Conda build 3 features are explained here: https://conda.io/docs/user-guide/tasks/build-packages/variants.html

Now that we have dependencies compiled. Here is how the CI will work in Bob.

• provide a conda channel for stable bob packages and its dependencies.
  • provide a conda channel for beta bob packages and its dependencies. #49
• bob.conda will build only bob dependencies and not bob packages itself. bob.conda#27
• Create recipes for the rest of bob packages that are tested in bob.nightlies bob.conda#28
  • Provide instructions and scripts to create conda recipes for new packages. bob.conda#29
  • Put recipes in a folder called <package>.recipe in the project repository. bob.conda#27
  • The recipes would take some options from the environment variables. (like pinnings) not needed with conda build 3 see bob.conda!208
  • The pinnings and environments would be maintained here in bob.admin.
  • A script will be used to keep the pinnings updated in all repos. This encourages reproducibility.
• Make sure the build and test phase are done with conda packages. (no compiling using buildout)
• Publish changes from tags to the stable channel
  • Running the pipeline again for tag would increase the build number and publish a new package without replacing the old one.
    • When a new package for a stable release is required (maybe pinnings has changed) the whole pipeline must run again and not just the deploy step. This will make sure that the package is compiled again with the new requirements.
  • Build number should be taken from the environment. This would be 0 if the package has not been uploaded yet but would increase if there are already
  • The deployment script should not fail if it is run again (like if the package already exists in PyPI). some packages with the same version.
• Publish changes from master to the beta channel
  • Any new pipeline would replace the beta package. (conda has no concept of update packages unless their name actually changes so we need to increase the build number of beta packages too but remove the old ones to keep storage size intact.)
• bob.nightlies would update all beta conda packages.
• Deprecate conda-forge packages and its deployment scripts.
• Tip of projects (while being in beta) to be tested against stable packages with an option to fail.
• We need to have a nightlies for stable packages. To make sure all stable ones are working against each other.
• We need to include Sphinx-built docs with the package on the directory share/doc
• Add support for parsing conda build recipes in bob.extension bob.extension#38 (closed)
• Create a test package like toolchain that would export some useful variables and maybe depends on toolchain too. bob.conda#30