Overhauling our CI and pipelines
I'm proposing we should re-think the whole CI procedure entirely. Our CI system was made for the time we had some C++ code to be compiled and made available for other packages (@lcolbois: this was probably another reason for the splits).
As of now and thanks to a lot of work by predecessors, most of the packages in this group are Python-only - this is certainly true for the (biometrics) nightlies. This means we can probably benefit from this fact and improve our pipelines, at least for most packages in the bob/beat namespace.
I have a proposal in mind, but I think we need to debate this a bit more thoroughly as I may not be thinking about all aspects and oversimplifying. The bulk of the proposal goes through revising our pipeline requirements and moving most of the packages into a python-only workflow (for most part). The proposed pipeline would be like this:
- QA via
pre-commit
, if.pre-commit-config.yaml
is present - only linux, should fail fast. Caching is enabled. - Test (via pytest) on various platforms and python versions - uses a Python docker image (not conda), respects python package pinning, retrieve beta versions from internal Gitlab package registry (if they exist), otherwise PyPI. Pip caching is enabled.
- Generate sphinx documentation building and doctests (if
doc/
directory is present) - only linux, via Python docker image (not conda) - Package python code (via
python setup.py sdist
) - Package conda package using the sdist produced at the previous step, run tests (single platform - linux, just to cross-check)
- Deploy sdist (unreleased beta) package on internal "group" package registry (GitLab: https://docs.gitlab.com/ee/user/packages/pypi_repository/) if not tagged or private/internal. Deploy sdist package on PyPI if tagged and public.
- Deploy conda package on internal beta channel if not tagged, deploy on stable channel if tagged.
- Deploy documentation and coverage information on DAV web server if success (master and stable, if required)
For non-python packages, maintain minimal CI configuration, but allow maximum flexibility via their own personalised conda_build_config.yaml
. Everything else should match the above pipeline.
I also have some ideas on how to improve the packaging:
- Moving to
pyproject.toml
(https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html), and - Reflect that automatically on conda packages to deduplicate package lists (https://docs.conda.io/projects/conda-build/en/stable/resources/define-metadata.html#loading-data-from-other-files)
I expect this plan should make our pipelines run much faster, while simplifying the setup and testing. Conda environment creation continues to be supported and possible. For other simpler setups, simple Python software management (e.g. via pip or poetry) would be possible.
(This would affect issue #102 (closed) for example, #104 (closed) and maybe #103.)