bob.pipelines merge requests
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests
2020-04-08T16:09:19Z
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/21
Adaptive and heterogenous resource allocation
2020-04-08T16:09:19Z
Tiago de Freitas Pereira
Adaptive and heterogenous resource allocation
Hi people
With this MR, we can submit jobs via dask SpecCluster with different SGE specifications and I allowed the scheduler to **dynamically** request for some SGE job specs.
The Dask scheduler has a heuristics to create and dest...
Hi people
With this MR, we can submit jobs via dask SpecCluster with different SGE specifications and I allowed the scheduler to **dynamically** request for some SGE job specs.
The Dask scheduler has a heuristics to create and destroy workers based on CPU and MEMORY load.
The scheduler that I extended does the same (it's the same method), but also request for SGE requirements if necessary.
Follow below a use case.
The code below executes the following graph:
![Screen_Shot_2020-04-06_at_12.06.51](/uploads/fb4488da28c43dce35e0d659ff5d766c/Screen_Shot_2020-04-06_at_12.06.51.png)
All the jobs can be executed in `all.q` hosts, except the bottleneck one (pointed in the image).
For this one, only `q_1day` hosts can take it.
Our scheduler (`IdiapScheduler`) recognizes that a task requests an specific type of worker and just ask it.
Our `SGEIdiapCluster` recognizes that kind of requests and does `qsub` with the proper requirements
```python
# from bob.pipelines.distributed.sge import SGEIdiapCluster
from dask.distributed import Client
from bob.pipelines.distributed.sge import SGEIdiapCluster, AdaptiveIdiap
from dask.distributed import Client
import time
import os
import logging
logger = logging.getLogger("distributed")
logger.setLevel(0)
def dependency_example(client):
import dask
# Defining dummy jobs
def inc(x, value):
time.sleep(0.25)
return x + value
def bottleneck(x):
time.sleep(20)
return sum(x)
# Defining my graph
X = list(range(100))
#X = list(range(2))
output = []
resources = dict()
paralell_data = []
for x in X:
a = dask.delayed(inc)(x, 1)
b = dask.delayed(inc)(a, 2)
paralell_data.append(b)
# Bottleneck
c = dask.delayed(bottleneck)(paralell_data)
# LOOK HERE
# Annotating certain jobs to run in the GPU
resources[tuple(c.__dask_keys__())] = {'q_1day':1}
final_parelell_data = []
for x in paralell_data:
final_parelell_data.append(dask.delayed(inc)(x, c))
total = dask.delayed(sum)(final_parelell_data)
print(total.compute(scheduler=client, resources=resources))
#print(total.compute(scheduler="single-threaded"))
Q_1DAY_ALL = {
"q_1day": {
"queue": "q_1day",
"memory": "8GB",
"io_big": True,
"resource_spec": "",
"resources": {"q_1day":1},
},
"default": {
"queue": "all.q",
"memory": "4GB",
"io_big": False,
"resource_spec": "",
"resources": "",
}
}
cluster = SGEIdiapCluster(sge_job_spec=Q_1DAY_ALL)
cluster.scale(1, sge_job_spec_key="default")
cluster.adapt(minimum=0, maximum=48)
client = Client(cluster)
dependency_example(client)
#dependency_example(None)
```
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/22
New User Guide
2020-04-22T15:59:33Z
Tiago de Freitas Pereira
New User Guide
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/23
Better defaults for .adapt method
2020-04-23T10:43:15Z
Tiago de Freitas Pereira
Better defaults for .adapt method
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/24
Queues defaults
2020-04-24T08:28:46Z
Tiago de Freitas Pereira
Queues defaults
Approached the points discussed yesterday.
- Renamed SGEIdiapCluster to SGEMultipleQueuesCluster
- Renamed AdaptiveIdiaoQueue to AdaptiveMultipleQueues
- Set 2 queues setups containing all Idiap Queues -- https://gitlab.idiap.ch/bob/...
Approached the points discussed yesterday.
- Renamed SGEIdiapCluster to SGEMultipleQueuesCluster
- Renamed AdaptiveIdiaoQueue to AdaptiveMultipleQueues
- Set 2 queues setups containing all Idiap Queues -- https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/85476238efc404202b67945e2f89a0697e2579c4/bob/pipelines/distributed/sge_queues.py
- Turning `SGEMultipleQueuesCluster` Adaptive by default
ping @andre.anjos @amohammadi
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/25
Changed function name for testing is an object is picklable
2020-04-30T11:12:26Z
Tiago de Freitas Pereira
Changed function name for testing is an object is picklable
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/26
Refactor Mixins to Aggregators
2020-05-01T17:10:53Z
Amir MOHAMMADI
Refactor Mixins to Aggregators
Fixes #11
Fixes #11
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/27
Add tests for transformers
2020-05-02T13:22:57Z
Amir MOHAMMADI
Add tests for transformers
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/28
Filelist datasets
2020-05-14T08:27:07Z
André Anjos
Filelist datasets
@tiago.pereira, @ydayer: here is the CSV and JSON implementations of **filelist**-based datasets I had in my package, for your review.
The way the API goes makes sense for an application scenario in which loading an individual sample is...
@tiago.pereira, @ydayer: here is the CSV and JSON implementations of **filelist**-based datasets I had in my package, for your review.
The way the API goes makes sense for an application scenario in which loading an individual sample is costly (e.g. sample data is stored on disk).
If the sample data is not stored on disk (e.g., a dataset that can be completely stored in a single CSV table), then using other techniques would be better (e.g. pandas data frames).
I hope it is useful.
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/29
Fixed mario.wrap([dask]). It was not possible to set the npartitions kwarg
2020-05-02T12:53:30Z
Tiago de Freitas Pereira
Fixed mario.wrap([dask]). It was not possible to set the npartitions kwarg
```python
mario.wrap(["dask"],pipeline, npartitions=n)
```
wasn't working.
```python
mario.wrap(["dask"],pipeline, npartitions=n)
```
wasn't working.
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/30
Add DatasetPipeline to work on xarray datasets
2020-05-22T19:25:15Z
Amir MOHAMMADI
Add DatasetPipeline to work on xarray datasets
Fixes #15
Fixes #15
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/31
Make a sampleset work transparently with list of DelayedSamples
2020-05-25T08:48:52Z
Tiago de Freitas Pereira
Make a sampleset work transparently with list of DelayedSamples
This makes us able to make
`SampleSet([Sample(1),Sample(2)])` equals to `SampleSet(DelayedSample(load_function))`
where `load_function` loads `[Sample(1),Sample(2)]`
This makes us able to make
`SampleSet([Sample(1),Sample(2)])` equals to `SampleSet(DelayedSample(load_function))`
where `load_function` loads `[Sample(1),Sample(2)]`
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/32
Created a method that emulates isinsntace for aggregation
2020-06-18T12:36:15Z
Tiago de Freitas Pereira
Created a method that emulates isinsntace for aggregation
This will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something else
This will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something else
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/33
SGE defaults
2020-07-07T11:50:31Z
Tiago de Freitas Pereira
SGE defaults
Most of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.
Most of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/34
Project variable
2020-07-24T10:59:20Z
Tiago de Freitas Pereira
Project variable
Added `-P <project>` to the SGECluster
Added `-P <project>` to the SGECluster
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/35
Add DelayedSampleSet, remove bob.pipelines script, add pre-commit
2020-07-24T12:23:40Z
Amir MOHAMMADI
Add DelayedSampleSet, remove bob.pipelines script, add pre-commit
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/36
For some reason, the class information is not passed in the sample wrapper
2020-08-31T10:13:47Z
Tiago de Freitas Pereira
For some reason, the class information is not passed in the sample wrapper
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/37
Revert "For some reason, the class information is not passed in the sample wr...
2020-10-06T16:43:02Z
Amir MOHAMMADI
Revert "For some reason, the class information is not passed in the sample wrapper"
This reverts merge request !36
This reverts merge request !36
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/38
[sge] In dask some sublacessd classes need a config name. Fixes #20
2020-10-12T11:25:49Z
Amir MOHAMMADI
[sge] In dask some sublacessd classes need a config name. Fixes #20
Bob 9.0.0
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/39
[dask][sge] Added the variables `idle_timeout` and `allowed_failures` as
2020-10-12T15:04:50Z
Tiago de Freitas Pereira
[dask][sge] Added the variables `idle_timeout` and `allowed_failures` as
part of our `.bobrc` and added better defaults
part of our `.bobrc` and added better defaults
Bob 9.0.0
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/40
Add dask-client configurations as resources
2020-10-12T14:19:51Z
Yannick DAYER
Add dask-client configurations as resources
Fixes #19
Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.
Depends on bob.bio.base!201
Fixes #19
Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.
Depends on bob.bio.base!201
Bob 9.0.0
Yannick DAYER
Yannick DAYER