Skip to content

ArcFace (MXNET) + IJBC lead to memory error

Hi, For info I tried to run the ArcFace baseline on IJBC but I can't seem to make it work, there is a memory error like

mxnet.base.MXNetError: [11:11:36] /tmp/build/80754af9/libmxnet_1564766659613/work/src/storage/./cpu_device_storage.h:75: Failed to allocate CPU Memory
Stack trace:
  [bt] (0) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x38e1c4) [0x7f9ad17381c4]
  [bt] (1) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x26a34f3) [0x7f9ad3a4d4f3]
  [bt] (2) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*)+0x5d) [0x7f9ad3a52dbd]
  [bt] (3) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x399cd1) [0x7f9ad1743cd1]
  [bt] (4) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x493357) [0x7f9ad183d357]
  [bt] (5) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x496107) [0x7f9ad1840107]
  [bt] (6) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49653e) [0x7f9ad184053e]
  [bt] (7) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49da1d) [0x7f9ad1847a1d]
  [bt] (8) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49dadf) [0x7f9ad1847adf]

I tried to play a bit with the Dask partition size parameter with no success. Note that the ArcFace model works for other lighter databases (Mobio, Multipie), and IJBC works with TF2 baselines. But the MXNet + IJBC combination seems to be too demanding. I remember that we adjusted the TF2 implementation to add an option to toggle samplewise inference (vs by batch) to limit the memory footprint, is there something similar we should do with MXNet models ?

I can manage my work without running this evaluation, but I just wanted to point it out.