bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2020-11-22T18:40:31Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/48Improvements on CheckpointWrapper2020-11-22T18:40:31ZTiago de Freitas PereiraImprovements on CheckpointWrapperAdded the optional argument `hash_fn` in the `CheckpointWrapper` class.
Once this is set, `sample.key` generates a hash code and this hash code is used to compose the final path where `sample` will be checkpointed.
This is optional and ...Added the optional argument `hash_fn` in the `CheckpointWrapper` class.
Once this is set, `sample.key` generates a hash code and this hash code is used to compose the final path where `sample` will be checkpointed.
This is optional and generic enough for our purposes.
This hash function can be shipped in the database interface.
Closes https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/25Bob 9.0.0Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/47Multiple changes2020-11-23T10:27:21ZAmir MOHAMMADIMultiple changes* [DelayedSample] Allow for arbitrary delayed attributes
* [SampleBatch] Allow other attributes than data
Fixes #26 #24* [DelayedSample] Allow for arbitrary delayed attributes
* [SampleBatch] Allow other attributes than data
Fixes #26 #24Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/49[DelayedSample] Fix issues when an attribute was set2020-11-23T14:19:57ZAmir MOHAMMADI[DelayedSample] Fix issues when an attribute was setBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/50[DelayedSample(Set)] make load and delayed_attributes private2020-11-23T16:24:44ZAmir MOHAMMADI[DelayedSample(Set)] make load and delayed_attributes privateThis removes the need for a lot of guessing in downstream packages as they can start removing all keys that start with `_` when access of the sample's attribute is needed.This removes the need for a lot of guessing in downstream packages as they can start removing all keys that start with `_` when access of the sample's attribute is needed.Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/51[dask][sge] Multiqueue updates2020-11-25T16:51:11ZTiago de Freitas Pereira[dask][sge] Multiqueue updatesIn this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "chec...In this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path=model_path,
fit_tag="q_short_gpu",
)
```
You have to explicitly set the list of resource tags available.
```python
pipeline.fit_transform(...).compute(
scheduler=dask_client, resources=cluster.get_sge_resources()
```Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/53Updates2020-11-26T15:13:57ZTiago de Freitas PereiraUpdatesImplemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Implemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/52[CheckpointWrapper] Allow custom save and load functions through estimator tags2020-11-27T11:17:16ZAmir MOHAMMADI[CheckpointWrapper] Allow custom save and load functions through estimator tagsBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/54Fixed multiqueue2020-11-27T15:34:52ZTiago de Freitas PereiraFixed multiqueueHi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in t...Hi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in the same
way as before and fetch the resources like this
```python
pipeline = bob.pipelines.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path="./",
transform_extra_arguments=(("metadata", "metadata"),),
fit_tag="q_short_gpu",
)
from bob.pipelines.distributed.sge import get_resource_requirements
resources = get_resource_requirements(pipeline)
pipeline.fit_transform(X_as_sample).compute(
scheduler=client, resources=resources
)
```https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/55Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines2020-12-09T10:15:36ZTiago de Freitas PereiraMoved VALID_DASK_CLIENT_STRINGS to bob.pipelineshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/56Two new features2020-12-07T13:08:49ZTiago de Freitas PereiraTwo new features- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-up- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-uphttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/58Moved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a g...2020-12-08T16:24:03ZTiago de Freitas PereiraMoved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a general functionhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/59Dask client names2020-12-11T18:50:32ZAmir MOHAMMADIDask client namesBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60CSVSampleLoaders as transformers2020-12-14T13:59:48ZTiago de Freitas PereiraCSVSampleLoaders as transformersMade CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read ann...Made CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read annotations using `bob.db.base.read_anno...`.
This is delayed.
I'm already porting this stuff on `bob.bio.base`. Code is way more cleaner.
ping @amohammadi @ydayer
Closes #30 Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/61Fixed modules2021-02-02T10:44:41ZTiago de Freitas PereiraFixed modulesconfig files from here are not available once `conda install bob.pipelines`config files from here are not available once `conda install bob.pipelines`https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/62Implement a new simple generic csv-based database interface2021-03-22T16:13:46ZAmir MOHAMMADIImplement a new simple generic csv-based database interfaceDepends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126Depends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/63Implemented a mechanism in the Checkpoint wrapper that asserts if data was...2021-10-29T15:34:58ZTiago de Freitas PereiraImplemented a mechanism in the Checkpoint wrapper that asserts if data was...Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/64Fix delayed attributes2021-10-29T15:34:58ZYannick DAYERFix delayed attributesDelayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Delayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/66Handled failed processing (Failure to Acquire) in the wrappers2021-10-29T15:34:58ZYannick DAYERHandled failed processing (Failure to Acquire) in the wrappersFixes #32Fixes #32Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/67Some minor updates on the checkpoint wrapper and SGE2021-10-29T15:34:58ZTiago de Freitas PereiraSome minor updates on the checkpoint wrapper and SGEYannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/68Fix parent's delayed_attributes modified by child2021-10-29T15:34:58ZYannick DAYERFix parent's delayed_attributes modified by childA `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.A `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.Amir MOHAMMADIAmir MOHAMMADI