Skip to content
Snippets Groups Projects

WIP: Conda ci

Closed WIP: Conda ci
conda-ci into master
10 open threads
Closed
Amir MOHAMMADIrequested to merge
conda-ci into master
10 open threads

Ok I have configured some working CI here which uses small conda environments, caches things, and uses conda packages for our test process and upload process.

I have a couple of issues still:

  • upload of packages
    • Don't know how to upload the files like you did with wheels.
    • After a file is uploaded you need to run conda index to reflect the new changes.
  • the port process is somewhat explained here: https://gitlab.idiap.ch/bob/bob/wikis/ci-setup-with-conda but needs improvement.

ping @andre.anjos @tiago.pereira @mguenther @pkorshunov

Merge request reports

Pipeline #6138 passed

Stage: build
Stage: test

Pipeline passed for 07b424bf on conda-ci

Approval is optional

Closed by avatar (Aug 20, 2025 12:34am UTC)

Merge details

  • The changes were not merged into master.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
.conda-recipe/meta.yaml 0 → 100644
38 - libblitz
39 - numpy
40 - coverage
41 - sphinx
42
43 imports:
44 - bob
45 - bob.extension
46 - bob.extension.scripts
47
48 commands:
49 - bob_new_version.py --help
50 - bob_dependecy_graph.py --help
51 - bob_dbmanage.py all download --force || true
52 - python -c "from bob.extension import get_config; print(get_config())"
53 # - coverage run --source=bob.extension `which nosetests` -sv bob.extension # [not osx]
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • bootstrap-conda.sh 0 → 100755
    18 else
    19 ARCH="Linux-x86_64"
    20 fi
    21
    22 if [[ "$PYTHON_VERSION" == "2.7" ]]; then
    23 PYK="2"
    24 else
    25 PYK="3"
    26 fi
    27
    28 # Create root environment and add basic channels for conda
    29 if [ ! -x ${CONDA} ]; then
    30 if [ ! -x ${MINICONDA} ]; then
    31 mkdir -pv `dirname ${MINICONDA}`
    32 echo "[>>] Downloading `basename ${MINICONDA}` -> ${MINICONDA}..."
    33 curl --silent https://repo.continuum.io/miniconda/Miniconda${PYK}-latest-${ARCH}.sh --output ${MINICONDA}
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on an old version of the diff
    • .gitlab-ci.yml
    13 11 - build
    14 12 - test
    15 13 - docs
    16 - wheels
    17
    14 - upload
    18 15
    19 16 # Global variables
    20 17 variables:
    21 CONDA_PREFIX: env
    18 TEST_CHANNEL: file://idiap/group/torch5spro/bob_master_conda_channel
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • .gitlab-ci.yml
    13 11 - build
    14 12 - test
    15 13 - docs
    16 - wheels
    17
    14 - upload
    18 15
    19 16 # Global variables
    20 17 variables:
    21 CONDA_PREFIX: env
    18 TEST_CHANNEL: file://idiap/group/torch5spro/bob_master_conda_channel
    22 19
    20 # Global cache
    21 cache:
    22 key: "$CI_PROJECT_NAMESPACE/$CI_BUILD_NAME" # per group name and build name
  • Amir MOHAMMADI Added 1 commit:

    Added 1 commit:

    • e05c599b - [ci] remove git clean -ffdx
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • .gitlab-ci.yml
    27 29 stage: build
    28 30 before_script:
    29 31 - git clean -ffdx
    30 - curl --silent https://gitlab.idiap.ch/bob/bob/snippets/7/raw | tr -d '\r' > bootstrap-conda.sh
    31 - chmod 755 ./bootstrap-conda.sh
    32 - ./bootstrap-conda.sh ${CONDA_FOLDER} ${PYTHON_VER} ${CONDA_PREFIX}
    32 - ./bootstrap-conda.sh miniconda ${PYTHON_VER}
    33 33 variables: &build_variables
    34 34 BOB_DOCUMENTATION_SERVER: "http://www.idiap.ch/software/bob/docs/latest/bob/%s/master/"
    35 35 script:
    36 - ./bin/buildout
    37 - ./bin/sphinx-build doc sphinx
    38 - ./bin/python setup.py bdist_wheel --python-tag ${WHEEL_TAG}
    39 after_script:
    40 - rm -rf ${CONDA_PREFIX}
    36 - miniconda/bin/conda build --no-test --python=${PYTHON_VER} .conda-recipe
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • .gitlab-ci.yml
    28 30 before_script:
    29 31 - git clean -ffdx
    30 - curl --silent https://gitlab.idiap.ch/bob/bob/snippets/7/raw | tr -d '\r' > bootstrap-conda.sh
    31 - chmod 755 ./bootstrap-conda.sh
    32 - ./bootstrap-conda.sh ${CONDA_FOLDER} ${PYTHON_VER} ${CONDA_PREFIX}
    32 - ./bootstrap-conda.sh miniconda ${PYTHON_VER}
    33 33 variables: &build_variables
    34 34 BOB_DOCUMENTATION_SERVER: "http://www.idiap.ch/software/bob/docs/latest/bob/%s/master/"
    35 35 script:
    36 - ./bin/buildout
    37 - ./bin/sphinx-build doc sphinx
    38 - ./bin/python setup.py bdist_wheel --python-tag ${WHEEL_TAG}
    39 after_script:
    40 - rm -rf ${CONDA_PREFIX}
    36 - miniconda/bin/conda build --no-test --python=${PYTHON_VER} .conda-recipe
    37 - miniconda/bin/conda install -n root --yes ${CI_PROJECT_NAME}
  • Amir MOHAMMADI Added 1 commit:

    Added 1 commit:

    • 114c8ab1 - [ci] remove miniconda.sh from artifacts
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • .conda-recipe/meta.yaml 0 → 100644
    1 {% set version = "master" %}
    2
    3 package:
    4 name: bob.extension
    5 version: {{ version }}
    6
    7 source:
    8 git_url: ../
    9
    10 build:
    11 entry_points:
    12 - bob_new_version.py = bob.extension.scripts:new_version
    13 - bob_dependecy_graph.py = bob.extension.scripts:dependency_graph
    14 number: 0
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • bootstrap-conda.sh 0 → 100755
    29 if [ ! -x ${CONDA} ]; then
    30 if [ ! -x ${MINICONDA} ]; then
    31 mkdir -pv `dirname ${MINICONDA}`
    32 echo "[>>] Downloading `basename ${MINICONDA}` -> ${MINICONDA}..."
    33 curl --silent https://repo.continuum.io/miniconda/Miniconda${PYK}-latest-${ARCH}.sh --output ${MINICONDA}
    34 chmod 755 ${MINICONDA}
    35 fi
    36
    37 echo "[>>] Creating root environment and setting basic options..."
    38 bash ${MINICONDA} -b -p ${BASEDIR}
    39 fi
    40
    41 echo "[>>] Updating conda in the root environment..."
    42 touch ${BASEDIR}/.condarc
    43 ${CONDA} config --set show_channel_urls True
    44 ${CONDA} install --override-channels -c defaults --yes --quiet -n root conda-build sphinx sphinx_rtd_theme
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • bootstrap-conda.sh 0 → 100755
    36
    37 echo "[>>] Creating root environment and setting basic options..."
    38 bash ${MINICONDA} -b -p ${BASEDIR}
    39 fi
    40
    41 echo "[>>] Updating conda in the root environment..."
    42 touch ${BASEDIR}/.condarc
    43 ${CONDA} config --set show_channel_urls True
    44 ${CONDA} install --override-channels -c defaults --yes --quiet -n root conda-build sphinx sphinx_rtd_theme
    45 ${CONDA} update --override-channels -c defaults --yes --quiet -n root conda
    46 ${CONDA} config --add channels conda-forge
    47 ${CONDA} config --add channels defaults
    48 ${CONDA} config --add channels ${TEST_CHANNEL}
    49 ${CONDA} info
    50
    51 # Function for running command and echoing results
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on the diff
    • .conda-recipe/meta.yaml 0 → 100644
    9
    10 build:
    11 entry_points:
    12 - bob_new_version.py = bob.extension.scripts:new_version
    13 - bob_dependecy_graph.py = bob.extension.scripts:dependency_graph
    14 number: 0
    15 skip: true # [win]
    16 script_env:
    17 - CFLAGS
    18 - CXXFLAGS
    19 - LDFLAGS # [osx]
    20 - MACOSX_DEPLOYMENT_TARGET # [osx]
    21
    22 requirements:
    23 build:
    24 - python
  • Amir MOHAMMADI Added 1 commit:

    Added 1 commit:

    • 5427cef4 - [ci] test with docker image
  • André Anjos
    André Anjos @aanjos
    Owner

    Hello,

    I finally had the time to look into your suggestion. As of now, packages don't pin their dependencies.

    Pinning for all individual packages is probably not a good idea: every time we'll need to do a global update for version numbers, multiple package may have to be touched.

    This will potentially generate a lot of work. Or not?

  • André Anjos
    André Anjos @aanjos
    Owner

    Ideally, there should be a central authority with the pinnings for every individual possible (external) dependency. Individual packages should just respect that.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    The Idea is that we never update these pinnings.

  • André Anjos
    André Anjos @aanjos
    Owner

    Ok, why?

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    It's like testing on an old Linux distribution. We fix the versions and always test with these as our minimal supported version.

  • André Anjos
    André Anjos @aanjos
    Owner

    What about the support for API changes and the such on newer software? IOW, how to guarantee from our nightlies the builds will run on conda-forge?

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    You can't let packages be unpinned and you can't keep changing them all the time. What do you propose?

  • André Anjos
    André Anjos @aanjos
    Owner

    As of today, the packages compile against a cloned version of the environment. If I change the environment on the CIs, then all packages start to compile against the new environment. We had foreseen and implemented (thanks to @tiago.pereira), wheels uploading from the nightlies, what mitigated problems related to the environment changing (API and ABI compatibility issues), allowing us to re-compile the whole system in a single shot. The workflow for updating the environment was: update the CIs (this means 2 machines in the near future or 1 machine + 1 docker container), re-run the nightlies, done.

    Now, if you'd like to update the environment, what the workflow will be? From what you're proposing, it will be more like what we had before in which every package will need to be re-compiled in order. I can tell you it will take anyone doing it a few working days, even if they are experienced.

    Isn't it possible to, somehow, emulate the "clone" behaviour by caching our required packages on a web-server folder for example? Then, to update the environment, the work would be: update the packages on the web-folder, re-run the nightlies, done. In this way, no specific pinning would be required on the packages.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    So you are saying that you always want to test against the tip of conda-forge, am I right? And will not support old packages?

    Edited by Amir MOHAMMADI
  • André Anjos
    André Anjos @aanjos
    Owner

    Not exactly - I'm saying that we need to have flexibility to change the test environment w/o implying in a change in all packages.

  • André Anjos
    André Anjos @aanjos
    Owner

    Which environment we'll use for the CI is yet another story. In this corner, I don't care much about what we use if you tell me it is going to work OK in conda-forge.

  • André Anjos
    André Anjos @aanjos
    Owner

    One of our current difficulties is anticipating and fixing errors on that environment.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    In this corner, I don't care much about what we use if you tell me it is going to work OK in conda-forge.

    I don't know! To make sure it works you need to test against it.

    There is a script that can update the pinnings in all the packages automatically.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    The script is here: https://github.com/conda-forge/conda-forge.github.io/blob/master/scripts/pin_the_slow_way.py

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    You can also use environment variables in the meta.yaml files, like saying ffmpeg=$FFMPEG_VERSION

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    You can take a look at https://github.com/conda-forge/bob.io.image-feedstock/blob/master/recipe/meta.yaml for pinning example from environment variables.

  • André Anjos
    André Anjos @aanjos
    Owner

    Following an offline discussion, the current proposal is to have the versions explicitly stated only at the bootstrap-conda.sh on the bob/bob.admin package.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    Would you be kind and generate a list of packages that you download from conda-forge? I need to give it to them and ask them not to delete them in future. The links are in these files: https://gitlab.idiap.ch/bob/bob.admin/blob/master/linux/devel-py27.txt

    I would need a list like this:

    https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py27_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ffmpeg-2.8.6-2.tar.bz2
  • André Anjos
    André Anjos @aanjos
    Owner

    There is now a script in bob.admin to do that. If you clone that package and run it, it will produce the following file list:

    $ ./conda-forge-keeplist.sh 
https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py27_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py34_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py35_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ffmpeg-2.8.6-2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/giflib-5.1.2-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/icu-56.1-4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py27_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py34_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py35_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libblitz-0.10-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libmatio-1.5.6-4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libsvm-3.21-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/sox-14.4.2-libpng1.6.22_6.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/vlfeat-0.9.20-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/x264-20131217-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py27_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py34_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py35_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ffmpeg-2.8.6-2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/giflib-5.1.2-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/icu-56.1-4.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py27_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py34_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py35_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libblitz-0.10-0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libmatio-1.5.6-4.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libsvm-3.21-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/sox-14.4.2-libpng1.6.22_6.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/vlfeat-0.9.20-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/x264-20131217-1.tar.bz2
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    I have raised an issue here: https://github.com/conda-forge/conda-forge.github.io/issues/220 to keep track of these package deletions. Until then, can we have a backup of them locally?

  • André Anjos
    André Anjos @aanjos
    Owner

    They are on the disk on all CIs. Make a copy from there.

  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi
    Author Owner

    Writing a script that does these changes automatically would be great. An initial example script would be: https://github.com/conda-forge/conda-forge.github.io/blob/master/scripts/pin_the_slow_way.py

  • André Anjos
    André Anjos @aanjos
    Owner

    After issues we had with conda locking, do we still want this?

  • André Anjos added 1 commit

    added 1 commit

    Compare with previous version

  • André Anjos added 1 commit

    added 1 commit

    Compare with previous version

  • André Anjos added 1 commit

    added 1 commit

    Compare with previous version

  • Amir MOHAMMADI mentioned in issue bob.admin#35 (closed)

    mentioned in issue bob.admin#35 (closed)

  • Amir MOHAMMADI removed milestone

    removed milestone

  • Amir MOHAMMADI closed

    closed

  • Amir MOHAMMADI changed milestone to %Conda-based CI

    changed milestone to %Conda-based CI

    • Please register or sign in to reply