Nightlies (MacOSX) are failing because of this one

It's weird. What has changed?

Don't know.

I will have a look at it next week

The Seg Fault is being raised here https://gitlab.idiap.ch/bob/bob.learn.boosting/blob/ac051da72e16b65dce71a1e46a0925ed634a20fe/bob/learn/boosting/tests/test_boosting.py#L55

For some reason the StumpMachine is screwed. I can't get anything from this object after it's trained

machine.forward(inputs[0].astype("uint16")) works fine after the training

The problem seems deeper, all the trainers (StumpTrainer, Boosting, LUTTrainer) are dumping partially corrupted machines.

Need insights on this one

@tiago.pereira try to remove bob-devel from the conda recipe and run ../bob.admin/conda/conda-bootsrap.py boosting2 and see if you get the same error with this second conda environment.

Code snippet to reproduce the issue.

import bob.learn.boosting
weak = bob.learn.boosting.StumpMachine(0.1,1,0)
machine = bob.learn.boosting.BoostedMachine()
machine.add_weak_machine(weak, 0.1)
print(machine.weak_machines[0].polarity)

For some reason here:

https://gitlab.idiap.ch/bob/bob.learn.boosting/blob/ac051da72e16b65dce71a1e46a0925ed634a20fe/bob/learn/boosting/boosted_machine.cpp#L143

the std::vector is already "screwed" for macosx

self->base->getWeakMachines()

Guys,

follow the compilation line of the file BoostedMachine.cpp from the build that crashes:

/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_build_env/bin/x86_64-apple-darwin13.4.0-clang++  -DHAVE_BLITZ=1 -DHAVE_BOOST=1 -DHAVE_HDF5=1 -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API -Dbob_learn_boosting_EXPORTS -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/core/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/io/base/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/bob/learn/boosting/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/blitz/include -isystem /Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -isystem /Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/numpy/core/include  -std=c++0x -pedantic -Wall -pthread -Qunused-arguments -m64 -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/work=/usr/local/src/conda/bob.learn.boosting-2.0.16b0 -fdebug-prefix-map=/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.9 -fPIC   -o CMakeFiles/bob_learn_boosting.dir/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/bob/learn/boosting/cpp/BoostedMachine.cpp.o -c /Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/bob/learn/boosting/cpp/BoostedMachine.cpp

AND from the binds boosted_machine.cpp

x86_64-apple-darwin13.4.0-clang -DNDEBUG -fwrapv -O2 -Wall -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -Wno-strict-aliasing -DBOB_EXT_MODULE_PREFIX="bob.learn.boosting" -DBOB_EXT_MODULE_NAME="_library" -DBOB_EXT_ENTRY_NAME=PyInit__library -DBOB_EXT_MODULE_VERSION="2.0.16b0" -DHAVE_BOOST=1 -DHAVE_HDF5=1 -DHAVE_BLITZ=1 -DPY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/bob/learn/boosting/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/blitz/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/extension/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/core/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/bob/io/base/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -I/Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/python3.6m -c bob/learn/boosting/boosted_machine.cpp -o build/temp.macosx-10.9-x86_64-3.6/bob/learn/boosting/boosted_machine.o -std=c++0x -Wno-#warnings -pthread -isystem /Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/numpy/core/include -isystem /Users/gitlab/builds/b6d3167a/0/bob/bob.learn.boosting/miniconda/conda-bld/bob.learn.boosting_1541169394807/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include

I think this might be related with the flag -isysroot set in the compilation of BoostedMachine.cpp. It seems that we are building the C++ level with the SDK 10.13 and the binds with 10.9. Hence, we have an ABI problem.

If you observe the last successful build (https://gitlab.idiap.ch/bob/bob.learn.boosting/-/jobs/143762), we don't have that.

Looking at the diff an old build and a new build, looks like a new build of clang and its libraries are released.

@amohammadi and me tried to update the SDK in my laptop, but still we didn't manage to solve the issue.

@tiago.pereira can you try running buildout with buildout -c develop.cfg and 10.9 SDK? maybe one of our bob dependencies are compiled wrongly.

Still f****d

Do conda remove bob.extension and try again your test.

Looks like this -isysroot command appeared in nightlies in Oct 16th in: https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150633/raw

/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_build_env/bin/x86_64-apple-darwin13.4.0-clang++

-DHAVE_BLITZ=1
-DNO_IMPORT_ARRAY=1
-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION
-DPY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API
-Dbob_core_EXPORTS
-I/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/src/bob.core/bob/core/include
-I/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/blitz/include
-isystem
/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/numpy/core/include
/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-std=c++0x
-pedantic
-Wall
-pthread
-Qunused-arguments
-m64
-march=core2
-mtune=haswell
-mssse3
-ftree-vectorize
-fPIC
-fPIE
-fstack-protector-strong
-O2
-pipe
-stdlib=libc++
-fvisibility-inlines-hidden
-std=c++14
-fmessage-length=0
-I/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-fdebug-prefix-map=/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/work=/usr/local/src/conda/bob.core-2.2.2b0
-fdebug-prefix-map=/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/miniconda/conda-bld/bob.core_1539718013933/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place=/usr/local/src/conda-prefix
-D_FORTIFY_SOURCE=2
-mmacosx-version-min=10.9
-isysroot
/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
-o
CMakeFiles/bob_core.dir/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/src/bob.core/bob/core/cpp/logging.cpp.o
-c
/Users/gitlab/builds/b6d3167a/0/bob/bob.nightlies/src/bob.core/bob/core/cpp/logging.cpp

Didn't work and the thing didn't even build.

It's weird tho, it tried to link bob.core using the 10.14 SDK (I have the 10.13).

Guys, I can't focus on this. Would be nice to have someone in @bob to have a look in to this

This issue is most probably due to -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk being added to our build commands. This only happens when using cmake in a conda-build environment. I cannot reproduce this using python setup.py install or buildout.

The last build that did not have this issue was on 23rd of September! https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/148884

Nightlies (MacOSX) are failing because of this one

Designs

Child items ...

Activity