gridtk: ERROR: Job 'score-B-dev' (32 (2)) finished with status 'executing' instead of 'success' or 'failure'
I had a pipeline running for 88 minutes and it failed because the machine was slow (I think). See https://gitlab.idiap.ch/bob/bob/-/jobs/132232
======================================================================
FAIL: bob.bio.gmm.test.test_scripts.test_gmm_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/base/test/utils.py", line 62, in wrapper
return test(*args, **kwargs)
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/gmm/test/test_scripts.py", line 122, in test_gmm_parallel
_verify(parameters, test_dir, 'test_gmm_parallel', executable=main, ref_modifier='-gmm')
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/gmm/test/test_scripts.py", line 31, in _verify
assert os.path.exists(score_files[0]), "Score file %s does not exist" % score_files[0]
AssertionError: Score file /tmp/bobtest_1huix8r2/test_gmm_parallel/Default/nonorm/scores-dev does not exist
-------------------- >> begin captured logging << --------------------
gridtk: ERROR: Job 'enr-T-dev' (16 (2)) finished with status 'executing' instead of 'success' or 'failure'. Usually this means an internal error. Check your wrapper_script parameter!
gridtk: WARNING: Killing job '16 (1)' that was still running.
gridtk: WARNING: Killing job '16 (2)' that was still running.
--------------------- >> end captured logging << ---------------------
======================================================================
FAIL: bob.bio.gmm.test.test_scripts.test_ivector_lda_wccn_plda_parallel_missing_files
----------------------------------------------------------------------
Traceback (most recent call last):
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/base/test/utils.py", line 62, in wrapper
return test(*args, **kwargs)
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/gmm/test/test_scripts.py", line 379, in test_ivector_lda_wccn_plda_parallel_missing_files
_verify(parameters, test_dir, 'test_ivector_lda_wccn_plda_parallel', executable=main, ref_modifier='-ivector-lda-wccn-plda', allow_missing_files=True)
File "/local/builds/bob/bob/miniconda/conda-bld/bob_1523970986251/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/bob/bio/gmm/test/test_scripts.py", line 31, in _verify
assert os.path.exists(score_files[0]), "Score file %s does not exist" % score_files[0]
AssertionError: Score file /tmp/bobtest_xjsb2mew/test_ivector_lda_wccn_plda_parallel/Default/nonorm/scores-dev does not exist
-------------------- >> begin captured logging << --------------------
gridtk: ERROR: Job 'score-B-dev' (32 (2)) finished with status 'executing' instead of 'success' or 'failure'. Usually this means an internal error. Check your wrapper_script parameter!
gridtk: WARNING: Killing job '32 (1)' that was still running.
gridtk: WARNING: Killing job '32 (2)' that was still running.
--------------------- >> end captured logging << ---------------------