Multiproc data loading
Merge request reports
Activity
added 1 commit
- 95149ce4 - Changes on train/predict/experiment and test_cli files to add an option of...
- Resolved by André Anjos
@arenzom: everything should be fine now. Thank you. Set to auto-merge if build succeeds.
enabled an automatic merge when the pipeline for 96ae0825 succeeds
added 1 commit
- 6908db95 - [test.test_cli] Test mp with a single process (does not work on mac cis)
It might be related to this python issue. The default for multiprocess has changed from fork to spawn which may require some changes in the way resources are shared between parent and child processes.
If memory serves well, the default has changed with Python 3.8 but there still might be something related happening here.
More material: This SO thread indicates this problem is related to
pytorch==1.7.0
, which is the version we use. However, the thread also indicates the problem is reproducible on Python 3.6 and 3.8, whereas in our case, the problem only appears atPython==3.7+CI
.Edited by André AnjosThe fix here seems to work as a workaround for this problem.
added 1 commit
- 643ae030 - [script] Fix predict/train to use spawn context in case of multiprocess data loading