Skip to content

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Support
    • Submit feedback
    • Contribute to GitLab
  • Sign in
bob.bio.face
bob.bio.face
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Labels
    • Milestones
  • Merge Requests 3
    • Merge Requests 3
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • bob
  • bob.bio.facebob.bio.face
  • Issues
  • #42

Closed
Open
Opened Mar 01, 2021 by Laurent COLBOIS@lcolbois
  • Report abuse
  • New issue
Report abuse New issue

ArcFace (MXNET) + IJBC lead to memory error

Hi, For info I tried to run the ArcFace baseline on IJBC but I can't seem to make it work, there is a memory error like

mxnet.base.MXNetError: [11:11:36] /tmp/build/80754af9/libmxnet_1564766659613/work/src/storage/./cpu_device_storage.h:75: Failed to allocate CPU Memory
Stack trace:
  [bt] (0) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x38e1c4) [0x7f9ad17381c4]
  [bt] (1) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x26a34f3) [0x7f9ad3a4d4f3]
  [bt] (2) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*)+0x5d) [0x7f9ad3a52dbd]
  [bt] (3) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x399cd1) [0x7f9ad1743cd1]
  [bt] (4) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x493357) [0x7f9ad183d357]
  [bt] (5) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x496107) [0x7f9ad1840107]
  [bt] (6) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49653e) [0x7f9ad184053e]
  [bt] (7) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49da1d) [0x7f9ad1847a1d]
  [bt] (8) /idiap/temp/lcolbois/miniconda3/envs/bob_tf2/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x49dadf) [0x7f9ad1847adf]

I tried to play a bit with the Dask partition size parameter with no success. Note that the ArcFace model works for other lighter databases (Mobio, Multipie), and IJBC works with TF2 baselines. But the MXNet + IJBC combination seems to be too demanding. I remember that we adjusted the TF2 implementation to add an option to toggle samplewise inference (vs by batch) to limit the memory footprint, is there something similar we should do with MXNet models ?

I can manage my work without running this evaluation, but I just wanted to point it out.

Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
0
Labels
None
Assign labels
  • View project labels
Reference: bob/bob.bio.face#42