Skip to content
Snippets Groups Projects

Fixed multiqueue

Merged Tiago de Freitas Pereira requested to merge multi into master
1 unresolved thread

Hi @amohammadi @ydayer

I'm fixing here the issue raised with the multiqueue. I was wrongly setting all tasks to run in a particular resource restriction.

Now the problem is fixed.

To get it running you have to wrap your pipeline in the same way as before and fetch the resources like this

pipeline = bob.pipelines.wrap(
    ["sample", "checkpoint", "dask"],
    pipeline,
    model_path="./",
    transform_extra_arguments=(("metadata", "metadata"),),
    fit_tag="q_short_gpu",
)
from bob.pipelines.distributed.sge import get_resource_requirements
resources = get_resource_requirements(pipeline)

pipeline.fit_transform(X_as_sample).compute(
    scheduler=client, resources=resources
)

Merge request reports

Pipeline #46150 passed

Pipeline passed for 4b0bb706 on multi

Merged by Tiago de Freitas PereiraTiago de Freitas Pereira 4 years ago (Nov 27, 2020 3:34pm UTC)

Loading

Pipeline #46151 passed

Pipeline passed for ac67a380 on master

Deployed to be‎ta‎ 4 years ago

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
120 A :py:class`sklearn.pipeline.Pipeline` wrapper with :any:`bob.pipelines.DaskWrapper`
121
122 Example
123 -------
124 >>> cluster = SGEMultipleQueuesCluster(sge_job_spec=Q_1DAY_GPU_SPEC) # doctest: +SKIP
125 >>> client = Client(cluster) # doctest: +SKIP
126 >>> from bob.pipelines.sge import get_resource_requirements # doctest: +SKIP
127 >>> resources = get_resource_requirements(pipeline) # doctest: +SKIP
128 >>> my_delayed_task.compute(scheduler=client, resources=resources) # doctest: +SKIP
129 """
130
131 resources = dict()
132 for s in pipeline:
133 if hasattr(s, "resource_tags"):
134 resources.update(s.resource_tags)
135 return resources
  • Tiago de Freitas Pereira aborted the automatic merge because source branch was updated

    aborted the automatic merge because source branch was updated

  • added 1 commit

    Compare with previous version

  • added 1 commit

    • 4b0bb706 - Decreased the scale up/down delta checks

    Compare with previous version

  • mentioned in commit ac67a380

  • Please register or sign in to reply
    Loading