Segmentation fault in e step of ML GMM trainer
Here is code to reproduce:
import pkg_resources
import bob.io.base
import bob.learn.em
data = bob.io.base.load("data.hdf5")
with bob.io.base.HDF5File("machine.hdf5") as f:
machine = bob.learn.em.GMMMachine(f)
trainer = bob.learn.em.ML_GMMTrainer(True, True, True, 0)
trainer.initialize(machine)
trainer.e_step(machine, data)
(data and machine too large to upload here).
The data looks fine:
data.flags
Out[79]:
C_CONTIGUOUS : True
F_CONTIGUOUS : False
OWNDATA : True
WRITEABLE : True
ALIGNED : True
WRITEBACKIFCOPY : False
UPDATEIFCOPY : False
data.dtype
Out[80]: dtype('float64')
data.shape
Out[81]: (23716, 384)
data.min()
Out[82]: -11.06418228149414
data.max()
Out[83]: 10.025330543518066
data.mean()
Out[84]: 0.007353248348100698
data.std()
Out[85]: 2.3619958671051187
Looks like one the values becomes nan
in bob::math::Log::logAdd
:
Catchpoint 2 (exception thrown), __cxxabiv1::__cxa_throw (obj=obj@entry=0x5555560e1050, tinfo=0x7fffe94e07c0 <typeinfo for std::runtime_error>, dest=0x7fffe94331c8 <std::runtime_error::~runtime_error()>)
at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc:80
80 /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory.
(gdb) bt
#0 __cxxabiv1::__cxa_throw (obj=obj@entry=0x5555560e1050, tinfo=0x7fffe94e07c0 <typeinfo for std::runtime_error>, dest=0x7fffe94331c8 <std::runtime_error::~runtime_error()>)
at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc:80
#1 0x00007fffe85d9fc9 in bob::math::Log::logAdd (log_a=log_a@entry=-4026.0555379340872, log_b=log_b@entry=-nan(0x8000000000000)) at bob.learn.em/src/bob.math/bob/math/cpp/log.cpp:32
#2 0x00007fffe7bc5923 in bob::learn::em::GMMMachine::logLikelihood_ (this=this@entry=0x555556198c00, x=..., log_weighted_gaussian_likelihoods=...) at bob.learn.em/bob/learn/em/cpp/GMMMachine.cpp:247
#3 0x00007fffe7bcb3e6 in bob::learn::em::GMMMachine::logLikelihood (this=this@entry=0x555556198c00, x=..., log_weighted_gaussian_likelihoods=...) at bob.learn.em/bob/learn/em/cpp/GMMMachine.cpp:233
#4 0x00007fffe7bcc276 in bob::learn::em::GMMMachine::accStatistics (this=this@entry=0x555556198c00, x=..., stats=...) at bob.learn.em/bob/learn/em/cpp/GMMMachine.cpp:317
#5 0x00007fffe7bcc340 in bob::learn::em::GMMMachine::accStatistics (this=0x555556198c00, input=..., stats=...) at bob.learn.em/bob/learn/em/cpp/GMMMachine.cpp:294
#6 0x00007fffe7c2b4cb in bob::learn::em::GMMBaseTrainer::eStep (this=<optimized out>, gmm=..., data=...) at bob.learn.em/bob/learn/em/cpp/GMMBaseTrainer.cpp:41
#7 0x00007fffe7ac6a28 in bob::learn::em::ML_GMMTrainer::eStep (this=0x555555f4ad70, gmm=..., data=...) at bob.learn.em/bob/learn/em/include/bob.learn.em/ML_GMMTrainer.h:61
#8 0x00007fffe7ac7a76 in PyBobLearnEMMLGMMTrainer_e_step (self=0x7fffe9b93fb0, args=0x7fffe9b6ef88, kwargs=0x0) at bob/learn/em/ml_gmm_trainer.cpp:262
#9 0x0000555555665c54 in _PyCFunction_FastCallDict (func_obj=0x7fffe7ea59d8, args=<optimized out>, nargs=2, kwargs=0x0) at /tmp/build/80754af9/python_1564510748219/work/Objects/methodobject.c:231
#10 0x00005555556edabc in call_function (pp_stack=0x7fffffffc7c8, oparg=<optimized out>, kwnames=0x0) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4851
#11 0x000055555571075a in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335
#12 0x00005555556e89b9 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=0x0, argcount=<optimized out>,
args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=0x7ffff78d4a50) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166
#13 PyEval_EvalCodeEx (_co=0x7ffff78d4a50, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4187
#14 0x00005555556e975c in PyEval_EvalCode (co=co@entry=0x7ffff78d4a50, globals=globals@entry=0x7ffff7911318, locals=locals@entry=0x7ffff7911318) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:731
#15 0x0000555555769744 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff7911318, locals=0x7ffff7911318, flags=<optimized out>, arena=<optimized out>)
at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:1025
#16 0x0000555555769b41 in PyRun_FileExFlags (fp=0x555555987770, filename_str=<optimized out>, start=<optimized out>, globals=0x7ffff7911318, locals=0x7ffff7911318, closeit=1, flags=0x7fffffffca6c)
at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:978
#17 0x0000555555769d43 in PyRun_SimpleFileExFlags (fp=0x555555987770, filename=<optimized out>, closeit=1, flags=0x7fffffffca6c) at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:419
#18 0x000055555576d833 in run_file (p_cf=0x7fffffffca6c, filename=0x5555558cb0e0 L"test_em.py", fp=0x555555987770) at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:340
#19 Py_Main (argc=<optimized out>, argv=<optimized out>) at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:811
#20 0x000055555563788e in main (argc=2, argv=0x7fffffffcc78) at /tmp/build/80754af9/python_1564510748219/work/Programs/python.c:69