Parallelization issues when using ElasticDeformation makes training crash without any Exceptions or Errors
When training DenseNet with the standard config, which includes the ElasticDeformation augmentation, then the training will abruptly crash before it even properly starts IF --parallel is set to a positive value instead of the default value of -1 in the train.py script.
The issue is caused by the ElasticDeformation augmentation within the augmentations.py script. Specifically, the training will crash when the code reaches line 290:
self._mp_pool = multiprocessing.pool.Pool(instances)
This was temporarily fixable by changing the default "parallel" value of the ElasticDeformation class on line 260 in augmentations.py from -2 to -1:
def __init__(
self,
alpha: float = 1000.0,
sigma: float = 30.0,
spline_order: int = 1,
mode: str = "nearest",
p: float = 1.0,
parallel: int = -1, # When this is set to -2 as it currently is on the latest branch, it will crash
):EDIT: Apologies, the same code from line 290 is also on line 299, which could also have been the reason for this issue.
Activity
mentioned in commit 504b4749
mentioned in merge request !6 (merged)
@dcarron: This was fixed by @philip.abbet, and is correctly implemented in the main branch. However, please be attentive that !12 (merged) sets this to -1 presently! We need to revert that change before merging.
mentioned in commit 01e16fd7