Fix sub-packages dependency specification

This merge request aims to solve the issue identified in !461 (closed). The work there got a bit side-tracked trying to change to much of the recipe, thus this merge request which should tackle the issue directly.

Issue:
Currently build opencv package has an unsatisfiable dependency on py-opencv:
Problem: nothing provides py-opencv 4.5.0 py37h1209919_0 needed by opencv-4.5.0-py37_0
The py-opencv package that is build is the following:
py-opencv 4.5.0 py37h73dd053_0 bobconda.lab.idiap.ch/software/bob/conda/linux-64 1 MB which doesn't match the requirement from opencv.

Building the recipe locally with bdt build -vvv --stable -m conda_build_config.yaml conda/opencv would not reproduce the issue: the local opencv package depended on py-opencv with correct hash.

As far as I know the package hash is computed from the list of its dependencies which versions do matter for binary compatibility doc. Therefore by adding unused dependencies to py-opencv you can get a different hash for it, however the opencv dependency would still be unchanged.

Solution:
Issue appears to be that py-opencv dependency of opencv gets wrongfully pinned as the result of the wrong usage of the pin_subpackage function in run_exports sections of the subpackages. As stated in the doc:

• pin_subpackage should be used in run and/or test requirements to refer to a subpackage built by the parent recipe elsewhere in the recipe.

• run_exports sections are used to automatically put the package in the run-time dependency of any other package that has this package as a host and/or build dependency.

mentioned in merge request !461 (closed)

The opencv package needs to depend on the exact version since it is only a meta package keeping track of these 2 compatible packages (libopencv and py-opencv).

changed this line in version 2 of the diff

Right, it still works as intended locally.

added 1 commit

• 41c84867 - Pin subpackage exactly in opencv meta-package

Compare with previous version

@vpollet looking at the changes now, you have only changed the run_exports sections. I don't think this has any effect on our problem. If you want, I can merge this and we will see how it goes.

Yes let's try please, it looks like it solves the issue for me locally.

I think the issue came from using pin_subpackage in the run_export sections, which would somehow result in pinning the wrong version for opencv.

merged

mentioned in commit 74b4537c

Looks like you were right and it does not solve the issue.

I don't know what we should do to debug this further as I can't reproduce this behavior locally. The meta.yaml looks correct to me now.

Actually I might have the issue locally too, it's just that the build of the package is actually irrelevant and I get conflicts even if they match.

When building with bdt build -vvv --stable -m conda_build_config.yaml conda/opencv, at the end I get:

# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/libopencv-4.5.0-py37_0.tar.bz2
anaconda upload /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/py-opencv-4.5.0-py37h73dd053_0.tar.bz2
anaconda upload /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/opencv-4.5.0-py37_0.tar.bz2

and I can check that the dependencies look correct:

(temp) vpollet@italix03:/idiap/temp/vpollet/projects/batl/bob.conda2$ conda search --info /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/opencv-4.5.0-py37_0.tar.bz2
Loading channels: done
opencv 4.5.0 py37_0
-------------------
file name   : opencv-4.5.0-py37_0.tar.bz2
name        : opencv
version     : 4.5.0
build       : py37_0
build number: 0
size        : 24 KB
license     : BSD-3-Clause
subdir      : linux-64
url         : file:///idiap/temp/vpollet/anaconda3/conda-bld/linux-64/opencv-4.5.0-py37_0.tar.bz2
md5         : 8fa87be08ec6233cb280b04b3cc9d27e
timestamp   : 2020-12-03 16:07:41 UTC
dependencies: 
  - libopencv 4.5.0 py37_0
  - numpy >=1.16.6,<2.0a0
  - py-opencv 4.5.0 py37h73dd053_0
  - python >=3.7,<3.8.0a0


conda search --info /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/py-opencv-4.5.0-py37h73dd053_0.tar.bz2
Loading channels: done
py-opencv 4.5.0 py37h73dd053_0
------------------------------
file name   : py-opencv-4.5.0-py37h73dd053_0.tar.bz2
name        : py-opencv
version     : 4.5.0
build       : py37h73dd053_0
build number: 0
size        : 1.1 MB
license     : BSD-3-Clause
subdir      : linux-64
url         : file:///idiap/temp/vpollet/anaconda3/conda-bld/linux-64/py-opencv-4.5.0-py37h73dd053_0.tar.bz2
md5         : 99ef01e98876b135b53f53b138e05f28
timestamp   : 2020-12-03 16:06:37 UTC
dependencies: 
  - libopencv 4.5.0 py37_0
  - numpy >=1.16.6,<2.0a0
  - python >=3.7,<3.8.0a0

However if I want to use the packages I get a conflicts:

conda create -n temp
conda activate temp
conda install /idiap/temp/vpollet/anaconda3/conda-bld/linux-64/opencv-4.5.0-py37_0.tar.bz2
conda install --override-channels --strict-channel-priority -c http://www.idiap.ch/software/bob/conda -c defaults python=3.7

Collecting package metadata (current_repodata.json): done
Solving environment: - 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - local/linux-64::opencv==4.5.0=py37_0
failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: / 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - local/linux-64::opencv==4.5.0=py37_0
failed with initial frozen solve. Retrying with flexible solve.
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                                                                                            

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
opencv -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
opencv -> numpy[version='>=1.16.6,<2.0a0'] -> python[version='>=3.8,<3.9.0a0']
python=3.7
Note that strict channel priority may have removed packages required for satisfiability.

Any idea what is going on here ?

For the build in my previous message I had added python and numpy dependency to opencv output section like so:

  - name: opencv
    requirements:
      host:
        - python {{ python }}
        - numpy {{ numpy }}
      run:
        - python
        - {{ pin_compatible('numpy') }}
        - {{ pin_subpackage('libopencv', exact=True) }}
        - {{ pin_subpackage('py-opencv', exact=True) }}

so that's why they are listed in opencv's dependencies.

If python and numpy are not explicitely listed, then I get a similar conflict but opencv requires numpy=1.9.

changed the description

marked as a Work In Progress

unmarked as a Work In Progress

mentioned in commit 27824ec8

mentioned in merge request !465 (merged)