Skip to content

[hotfix] [Dask] Add configuration to access multithreaded queue

Laurent COLBOIS requested to merge sge-demanding into master

This adds an additional queue configuration that asks for multithreaded jobs, thus doubling the virtual memory limit to 16GB. With this fix I can run heavy Tensorflow baselines without OOM crash.

Long term it would be better to:

  • Expose the exact thread multiplier (now 2 by default)
  • Use the resource tagging of transformers to have the queue dynamically selected based on the transformer

Merge request reports