Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
bob.pipelines
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
bob
bob.pipelines
Merge requests
!87
Add a non-adaptive io-big queue
Code
Review changes
Check out branch
Download
Patches
Plain diff
Expand sidebar
Merged
Add a non-adaptive io-big queue
add-io-big
into
master
Overview
0
Commits
2
Pipelines
4
Changes
7
Merged
Add a non-adaptive io-big queue
Amir MOHAMMADI
requested to merge
add-io-big
into
master
Apr 6, 2022
Overview
0
Commits
2
Pipelines
4
Changes
7
0
0
Merge request reports
Compare
master
version 3
78c23425
Apr 25, 2022
version 2
623c38f3
Apr 6, 2022
version 1
4456ed53
Apr 6, 2022
master (base)
and
latest version
latest version
deb4c2c4
2 commits,
Apr 27, 2022
version 3
78c23425
1 commit,
Apr 25, 2022
version 2
623c38f3
1 commit,
Apr 6, 2022
version 1
4456ed53
1 commit,
Apr 6, 2022
7 files
+
112
−
20
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
7
bob/pipelines/config/distributed/sge_default.py
+
21
−
2
View file @ deb4c2c4
Edit in single-file editor
Open in Web IDE
from
dask.distributed
import
Client
from
bob.pipelines.distributed.sge
import
SGEMultipleQueuesCluster
from
bob.pipelines.distributed.sge
import
SGEMultipleQueuesCluster
,
get_max_jobs
from
bob.pipelines.distributed.sge_queues
import
QUEUE_DEFAULT
min_jobs
=
1
max_jobs
=
get_max_jobs
(
QUEUE_DEFAULT
)
cluster
=
SGEMultipleQueuesCluster
(
min_jobs
=
min_jobs
,
sge_job_spec
=
QUEUE_DEFAULT
)
cluster
.
scale
(
max_jobs
)
# Adapting to minimim 1 job to maximum 48 jobs
# interval: Milliseconds between checks from the scheduler
# wait_count: Number of consecutive times that a worker should be suggested for
# removal before we remove it.
cluster
.
adapt
(
minimum
=
min_jobs
,
maximum
=
max_jobs
,
wait_count
=
5
,
interval
=
10
,
target_duration
=
"
10s
"
,
)
cluster
=
SGEMultipleQueuesCluster
(
min_jobs
=
1
)
dask_client
=
Client
(
cluster
)
Loading