bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2021-10-29T15:34:58Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/75Add worker Time To Live limitation2021-10-29T15:34:58ZLaurent COLBOISAdd worker Time To Live limitationHello,
I have regularly been annoyed by Dask runs that hang indefinitely because of some workers being disconnected from the scheduler. In this case, the scheduler actually assumes the worker must still be doing its job so it doesn't rea...Hello,
I have regularly been annoyed by Dask runs that hang indefinitely because of some workers being disconnected from the scheduler. In this case, the scheduler actually assumes the worker must still be doing its job so it doesn't reassign the task, leading to a completely blocked run that needs to be interrupted by hand. This typically happens on very heavy experiments e.g. on IJBC, FRGC.
From what I understand this can be handled using the `worker_ttl` parameter of the scheduler, which puts a limit on how long a worker can be unseen by the scheduler before being killed and reassigning its task. It is `None` by default, I have been working for a while on a local branch where I set the default to 60s, it helped quite a lot.
I am proposing to merge this change, however I wanted to know what you think of it. My main concern is that it might be hiding some underlying issue (why do the workers actually disconnect ?), so I am not 100% sure it's a good change to make.
ping @tiago.pereira @amohammadihttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/64Fix delayed attributes2021-10-29T15:34:58ZYannick DAYERFix delayed attributesDelayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Delayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/74Add a DelayedSample.from_sample classmethod2021-10-29T15:34:58ZAmir MOHAMMADIAdd a DelayedSample.from_sample classmethodThis method can be used to transparently create new
DelayedSamples from either Samples or DelayedSamples without loading
delayed attributes and dataThis method can be used to transparently create new
DelayedSamples from either Samples or DelayedSamples without loading
delayed attributes and dataTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/73Remove samples_to_hdf5 methods2021-10-29T15:34:58ZAmir MOHAMMADIRemove samples_to_hdf5 methodsThese methods were not used anywhere.These methods were not used anywhere.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/72+ breakdown_SampleSet2021-10-29T15:34:58ZHatef OTROSHI+ breakdown_SampleSetAs desciribed in https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/33,
a function is added which takes as input a `SampleSet` with `N` samples and outputs `N` `SampleSets` with 1 `Sample` each.
ping @tiago.pereira
Closes #33 As desciribed in https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/33,
a function is added which takes as input a `SampleSet` with `N` samples and outputs `N` `SampleSets` with 1 `Sample` each.
ping @tiago.pereira
Closes #33 https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/71[CheckpointWrapper] Use atomic writing when saving features2021-10-29T15:34:58ZAmir MOHAMMADI[CheckpointWrapper] Use atomic writing when saving featuresTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/69[SampleSet] Do not load delayed attributes by not copying them over2021-10-29T15:34:58ZAmir MOHAMMADI[SampleSet] Do not load delayed attributes by not copying them overAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/68Fix parent's delayed_attributes modified by child2021-10-29T15:34:58ZYannick DAYERFix parent's delayed_attributes modified by childA `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.A `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/67Some minor updates on the checkpoint wrapper and SGE2021-10-29T15:34:58ZTiago de Freitas PereiraSome minor updates on the checkpoint wrapper and SGEYannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/66Handled failed processing (Failure to Acquire) in the wrappers2021-10-29T15:34:58ZYannick DAYERHandled failed processing (Failure to Acquire) in the wrappersFixes #32Fixes #32Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/63Implemented a mechanism in the Checkpoint wrapper that asserts if data was...2021-10-29T15:34:58ZTiago de Freitas PereiraImplemented a mechanism in the Checkpoint wrapper that asserts if data was...Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/62Implement a new simple generic csv-based database interface2021-03-22T16:13:46ZAmir MOHAMMADIImplement a new simple generic csv-based database interfaceDepends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126Depends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/61Fixed modules2021-02-02T10:44:41ZTiago de Freitas PereiraFixed modulesconfig files from here are not available once `conda install bob.pipelines`config files from here are not available once `conda install bob.pipelines`https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60CSVSampleLoaders as transformers2020-12-14T13:59:48ZTiago de Freitas PereiraCSVSampleLoaders as transformersMade CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read ann...Made CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read annotations using `bob.db.base.read_anno...`.
This is delayed.
I'm already porting this stuff on `bob.bio.base`. Code is way more cleaner.
ping @amohammadi @ydayer
Closes #30 Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/59Dask client names2020-12-11T18:50:32ZAmir MOHAMMADIDask client namesBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/55Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines2020-12-09T10:15:36ZTiago de Freitas PereiraMoved VALID_DASK_CLIENT_STRINGS to bob.pipelineshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/58Moved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a g...2020-12-08T16:24:03ZTiago de Freitas PereiraMoved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a general functionhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/56Two new features2020-12-07T13:08:49ZTiago de Freitas PereiraTwo new features- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-up- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-uphttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/54Fixed multiqueue2020-11-27T15:34:52ZTiago de Freitas PereiraFixed multiqueueHi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in t...Hi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in the same
way as before and fetch the resources like this
```python
pipeline = bob.pipelines.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path="./",
transform_extra_arguments=(("metadata", "metadata"),),
fit_tag="q_short_gpu",
)
from bob.pipelines.distributed.sge import get_resource_requirements
resources = get_resource_requirements(pipeline)
pipeline.fit_transform(X_as_sample).compute(
scheduler=client, resources=resources
)
```https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/52[CheckpointWrapper] Allow custom save and load functions through estimator tags2020-11-27T11:17:16ZAmir MOHAMMADI[CheckpointWrapper] Allow custom save and load functions through estimator tagsBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereira