bob.pipelines merge requests
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests
2020-08-31T10:13:47Z
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/36
For some reason, the class information is not passed in the sample wrapper
2020-08-31T10:13:47Z
Tiago de Freitas Pereira
For some reason, the class information is not passed in the sample wrapper
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/35
Add DelayedSampleSet, remove bob.pipelines script, add pre-commit
2020-07-24T12:23:40Z
Amir MOHAMMADI
Add DelayedSampleSet, remove bob.pipelines script, add pre-commit
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/34
Project variable
2020-07-24T10:59:20Z
Tiago de Freitas Pereira
Project variable
Added `-P <project>` to the SGECluster
Added `-P <project>` to the SGECluster
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/33
SGE defaults
2020-07-07T11:50:31Z
Tiago de Freitas Pereira
SGE defaults
Most of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.
Most of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/32
Created a method that emulates isinsntace for aggregation
2020-06-18T12:36:15Z
Tiago de Freitas Pereira
Created a method that emulates isinsntace for aggregation
This will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something else
This will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something else
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/31
Make a sampleset work transparently with list of DelayedSamples
2020-05-25T08:48:52Z
Tiago de Freitas Pereira
Make a sampleset work transparently with list of DelayedSamples
This makes us able to make
`SampleSet([Sample(1),Sample(2)])` equals to `SampleSet(DelayedSample(load_function))`
where `load_function` loads `[Sample(1),Sample(2)]`
This makes us able to make
`SampleSet([Sample(1),Sample(2)])` equals to `SampleSet(DelayedSample(load_function))`
where `load_function` loads `[Sample(1),Sample(2)]`
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/30
Add DatasetPipeline to work on xarray datasets
2020-05-22T19:25:15Z
Amir MOHAMMADI
Add DatasetPipeline to work on xarray datasets
Fixes #15
Fixes #15
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/29
Fixed mario.wrap([dask]). It was not possible to set the npartitions kwarg
2020-05-02T12:53:30Z
Tiago de Freitas Pereira
Fixed mario.wrap([dask]). It was not possible to set the npartitions kwarg
```python
mario.wrap(["dask"],pipeline, npartitions=n)
```
wasn't working.
```python
mario.wrap(["dask"],pipeline, npartitions=n)
```
wasn't working.
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/28
Filelist datasets
2020-05-14T08:27:07Z
André Anjos
Filelist datasets
@tiago.pereira, @ydayer: here is the CSV and JSON implementations of **filelist**-based datasets I had in my package, for your review.
The way the API goes makes sense for an application scenario in which loading an individual sample is...
@tiago.pereira, @ydayer: here is the CSV and JSON implementations of **filelist**-based datasets I had in my package, for your review.
The way the API goes makes sense for an application scenario in which loading an individual sample is costly (e.g. sample data is stored on disk).
If the sample data is not stored on disk (e.g., a dataset that can be completely stored in a single CSV table), then using other techniques would be better (e.g. pandas data frames).
I hope it is useful.
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/27
Add tests for transformers
2020-05-02T13:22:57Z
Amir MOHAMMADI
Add tests for transformers
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/26
Refactor Mixins to Aggregators
2020-05-01T17:10:53Z
Amir MOHAMMADI
Refactor Mixins to Aggregators
Fixes #11
Fixes #11
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/25
Changed function name for testing is an object is picklable
2020-04-30T11:12:26Z
Tiago de Freitas Pereira
Changed function name for testing is an object is picklable
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/24
Queues defaults
2020-04-24T08:28:46Z
Tiago de Freitas Pereira
Queues defaults
Approached the points discussed yesterday.
- Renamed SGEIdiapCluster to SGEMultipleQueuesCluster
- Renamed AdaptiveIdiaoQueue to AdaptiveMultipleQueues
- Set 2 queues setups containing all Idiap Queues -- https://gitlab.idiap.ch/bob/...
Approached the points discussed yesterday.
- Renamed SGEIdiapCluster to SGEMultipleQueuesCluster
- Renamed AdaptiveIdiaoQueue to AdaptiveMultipleQueues
- Set 2 queues setups containing all Idiap Queues -- https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/85476238efc404202b67945e2f89a0697e2579c4/bob/pipelines/distributed/sge_queues.py
- Turning `SGEMultipleQueuesCluster` Adaptive by default
ping @andre.anjos @amohammadi
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/23
Better defaults for .adapt method
2020-04-23T10:43:15Z
Tiago de Freitas Pereira
Better defaults for .adapt method
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/22
New User Guide
2020-04-22T15:59:33Z
Tiago de Freitas Pereira
New User Guide
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/21
Adaptive and heterogenous resource allocation
2020-04-08T16:09:19Z
Tiago de Freitas Pereira
Adaptive and heterogenous resource allocation
Hi people
With this MR, we can submit jobs via dask SpecCluster with different SGE specifications and I allowed the scheduler to **dynamically** request for some SGE job specs.
The Dask scheduler has a heuristics to create and dest...
Hi people
With this MR, we can submit jobs via dask SpecCluster with different SGE specifications and I allowed the scheduler to **dynamically** request for some SGE job specs.
The Dask scheduler has a heuristics to create and destroy workers based on CPU and MEMORY load.
The scheduler that I extended does the same (it's the same method), but also request for SGE requirements if necessary.
Follow below a use case.
The code below executes the following graph:
![Screen_Shot_2020-04-06_at_12.06.51](/uploads/fb4488da28c43dce35e0d659ff5d766c/Screen_Shot_2020-04-06_at_12.06.51.png)
All the jobs can be executed in `all.q` hosts, except the bottleneck one (pointed in the image).
For this one, only `q_1day` hosts can take it.
Our scheduler (`IdiapScheduler`) recognizes that a task requests an specific type of worker and just ask it.
Our `SGEIdiapCluster` recognizes that kind of requests and does `qsub` with the proper requirements
```python
# from bob.pipelines.distributed.sge import SGEIdiapCluster
from dask.distributed import Client
from bob.pipelines.distributed.sge import SGEIdiapCluster, AdaptiveIdiap
from dask.distributed import Client
import time
import os
import logging
logger = logging.getLogger("distributed")
logger.setLevel(0)
def dependency_example(client):
import dask
# Defining dummy jobs
def inc(x, value):
time.sleep(0.25)
return x + value
def bottleneck(x):
time.sleep(20)
return sum(x)
# Defining my graph
X = list(range(100))
#X = list(range(2))
output = []
resources = dict()
paralell_data = []
for x in X:
a = dask.delayed(inc)(x, 1)
b = dask.delayed(inc)(a, 2)
paralell_data.append(b)
# Bottleneck
c = dask.delayed(bottleneck)(paralell_data)
# LOOK HERE
# Annotating certain jobs to run in the GPU
resources[tuple(c.__dask_keys__())] = {'q_1day':1}
final_parelell_data = []
for x in paralell_data:
final_parelell_data.append(dask.delayed(inc)(x, c))
total = dask.delayed(sum)(final_parelell_data)
print(total.compute(scheduler=client, resources=resources))
#print(total.compute(scheduler="single-threaded"))
Q_1DAY_ALL = {
"q_1day": {
"queue": "q_1day",
"memory": "8GB",
"io_big": True,
"resource_spec": "",
"resources": {"q_1day":1},
},
"default": {
"queue": "all.q",
"memory": "4GB",
"io_big": False,
"resource_spec": "",
"resources": "",
}
}
cluster = SGEIdiapCluster(sge_job_spec=Q_1DAY_ALL)
cluster.scale(1, sge_job_spec_key="default")
cluster.adapt(minimum=0, maximum=48)
client = Client(cluster)
dependency_example(client)
#dependency_example(None)
```
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/20
Get SampleSet to accept DelayedSample
2020-04-06T06:25:28Z
Yannick DAYER
Get SampleSet to accept DelayedSample
Quick fix to accept both Sample and DelayedSample objects in
SampleSet.insert.
Fixes #9
Quick fix to accept both Sample and DelayedSample objects in
SampleSet.insert.
Fixes #9
Tiago de Freitas Pereira
Tiago de Freitas Pereira
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/19
Two updates
2020-04-03T07:02:37Z
Tiago de Freitas Pereira
Two updates
1. Created SampleSet.__len__ to count the samples inside of the sampleset
2. Created some logging in our main mixins
1. Created SampleSet.__len__ to count the samples inside of the sampleset
2. Created some logging in our main mixins
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/18
If there's no samples, returns an empty liss
2020-03-31T13:04:38Z
Tiago de Freitas Pereira
If there's no samples, returns an empty liss
https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/17
Using the same path used in the file checking in the delayed sample
2020-03-29T17:22:36Z
Tiago de Freitas Pereira
Using the same path used in the file checking in the delayed sample