Skip to content

[dask][sge] Multiqueue updates

Tiago de Freitas Pereira requested to merge multiqueue into master

In this merge request I:

  • Simplified the way multi-queue is set in our scripts
  • Updated our Dask documentation

Example

Setting the fit method to run on q_short_gpu

pipeline = mario.wrap(
    ["sample", "checkpoint", "dask"],
    pipeline,
    model_path=model_path,
    fit_tag="q_short_gpu",
)

You have to explicitly set the list of resource tags available.

pipeline.fit_transform(...).compute(
    scheduler=dask_client, resources=cluster.get_sge_resources()

Merge request reports

Loading