Skip to content
Snippets Groups Projects
Commit a3210f40 authored by Tiago de Freitas Pereira's avatar Tiago de Freitas Pereira
Browse files

[dask] Updated the target duration of a task to 10s. Being very aggressive in scale-up

parent 7b8e32c9
Branches
Tags
1 merge request!56Two new features
...@@ -290,10 +290,14 @@ class SGEMultipleQueuesCluster(JobQueueCluster): ...@@ -290,10 +290,14 @@ class SGEMultipleQueuesCluster(JobQueueCluster):
# interval: Milliseconds between checks from the scheduler # interval: Milliseconds between checks from the scheduler
# wait_count: Number of consecutive times that a worker should be suggested for # wait_count: Number of consecutive times that a worker should be suggested for
# removal before we remove it. # removal before we remove it.
# Here the goal is to wait 2 minutes before scaling down since
# it is very expensive to get jobs on the SGE grid
self.adapt(minimum=min_jobs, maximum=max_jobs, wait_count=5, interval=10) self.adapt(
minimum=min_jobs,
maximum=max_jobs,
wait_count=5,
interval=10,
target_duration="10s",
)
def _get_worker_spec_options(self, job_spec): def _get_worker_spec_options(self, job_spec):
"""Craft a dask worker_spec to be used in the qsub command.""" """Craft a dask worker_spec to be used in the qsub command."""
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment