Write a conda package for caffe_rtpose and bob.ip.rtpose

Hi @andre.anjos , @amohammadi , @pkorshunov

I would like to create a "conda stuff" package for

https://gitlab.idiap.ch/olivier.canevet/caffe_rtpose

and

https://gitlab.idiap.ch/bob/bob.ip.rtpose

Here are the commands to make it run locally on a machine (works on GPU at Idiap)

conda create -n env-for-rtpose
source activate env-for-rtpose

conda install -c conda-forge bob
conda install -c conda-forge cudatoolkit=7.5 cudnn=5.1
conda install -c conda-forge numpy opencv=2.4
conda install -c conda-forge protobuf gflags glog leveldb lmdb nose
conda install -c defaults hdf5
conda install scikit-image

cd /tmp
git clone -b conda https://gitlab.idiap.ch/olivier.canevet/caffe_rtpose
cd caffe_rtpose
make -j 12
make pycaffe

cd /tmp
git clone https://gitlab.idiap.ch/bob/bob.ip.rtpose


export PYTHONPATH=/tmp/caffe_rtpose/python:/tmp/bob.ip.rtpose

cd /tmp/bob.ip.rtpose/bob/ip/rtpose/test
python test_cpm.py

The results are

Could you help me?

Thanks

Olivier

https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/additional.html#conda-recipe and the readme of this package here.

I now get this error.

Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/opt/miniconda/bin/git', 'clone', '--mirror', 'http://gitlab.idiap.ch/bob/caffe_rtpose.git', '/local/builds/bob/bob.conda/conda-env/git_cache/gitlab.idiap.ch/bob/caffe_rtpose.git']' returned non-zero exit status 128
Traceback (most recent call last):
  File "/opt/miniconda/bin/conda-build-all", line 6, in <module>
    sys.exit(conda_build_all.cli.main())
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build_all/cli.py", line 90, in main
    b.main()
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 267, in main
    built_dist_location = self.build(meta, build_config)
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 204, in build
    conda_build.api.build(meta.meta, config=config)
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build/api.py", line 75, in build
    need_source_download=need_source_download, config=config)
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 1227, in build_tree
    config=recipe_config)
  File "/opt/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 791, in build
    assert not need_source_download, "Source download failed.  Please investigate."
AssertionError: Source download failed.  Please investigate.
WARNING: No ABI default set.  Falling back to compatibility mode with GCC 4.

It seems that the https://gitlab.idiap.ch/bob/caffe_rtpose.git is not accessible by the conda build process. How can I change that?

You need to go to https://gitlab.idiap.ch/bob/caffe_rtpose/settings/repository and in the deploy keys enable runner/beat-macosx and bpr-facedemo so that the CI machines can checkout your private source code.

It is explained for Bob packages in here: https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates#1-continuous-integration

My tests fail because of cublas_v2.h: No such file or directory.

make: *** [.build_release/src/caffe/layers/cudnn_pooling_layer.o] Error 1
In file included from ./include/caffe/common.hpp:19:0,
                 from ./include/caffe/blob.hpp:8,
                 from ./include/caffe/layers/hdf5_data_layer.hpp:9,
                 from src/caffe/layers/hdf5_data_layer.cpp:17:
./include/caffe/util/device_alternate.hpp:34:23: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
                       ^
compilation terminated.

Is it installed somewhere on the CI machines? No conda packages provide it as a source in conda/envs/<myenv>/include. Can you tell me the path of the installation on the server so that I can adapt the Makefile accordingly?

If cuda is not installed on the platform, the way to go, I think, would be to pack it as a conda package and have it hooked-in bob-devel (https://gitlab.idiap.ch/bob/bob-devel), so we can provide an uniform installation system.

Hi

I gave up writing a conda package for now. In the meantime, I provided instruction on https://gitlab.idiap.ch/bob/bob.ip.rtpose to install and run the code on GPU at Idiap. It uses the https://gitlab.idiap.ch/bob/caffe_rtpose that needs to be installed first. Instructions are https://gitlab.idiap.ch/bob/bob.ip.rtpose#installation-through-conda

If you want the https://gitlab.idiap.ch/bob/bob.ip.rtpose to be integrated properly in the bob framework/CI/buildout, we can meet so that you explain to me what the best solution is.

Hi @olivier.canevet you SHOULD NOT use the conda-forge channel. This is clearly mentioned in our installation instructions: https://www.idiap.ch/software/bob/install

Not only you shouldn't do that, you SHOULD NOT put conda-forge in installation instructions of one of Bob packages. This is serious. If people use conda-forge and our channel (or the defaults for that matter) at the same time, they WILL end up with wired ABI and API errors.

Are you using a particular package from conda-forge? If not, then just remove it. If yes, you can easily copy the recipe from conda-forge and add it to our channel.

If we are going to deprecate caffe (because of the lack of python-3.7 support), does this still make sense?

The license is not compatible, see https://gitlab.idiap.ch/bob/bob.ip.rtpose/issues/1

bob does not have a global license. Each package has its own license.

I don't know what this means. When Pavel used openpose (new version of rt_pose), there was a huge issue with license, and only the text output of the algo could be used, and nothing from openpose could be linked to bob.

I am not sure either but each bob package has a separate license. Some are BSD and some are GPLv3

Is this issue stale and interest is not there anymore?

@olivier.canevet: what about branch https://gitlab.idiap.ch/bob/bob.conda/tree/openpose can it be removed?

Yes, I think you can remove it because as far as I understand, "openpose" is research only. And caffe requires the some latest protobuf/lmdb/... packages, so we would probably build our own conda package for them.

Thx for your feedback. Closing.

closed

mentioned in commit 9123099a