bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2020-12-07T13:08:49Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/56Two new features2020-12-07T13:08:49ZTiago de Freitas PereiraTwo new features- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-up- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-uphttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/55Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines2020-12-09T10:15:36ZTiago de Freitas PereiraMoved VALID_DASK_CLIENT_STRINGS to bob.pipelineshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/54Fixed multiqueue2020-11-27T15:34:52ZTiago de Freitas PereiraFixed multiqueueHi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in t...Hi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in the same
way as before and fetch the resources like this
```python
pipeline = bob.pipelines.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path="./",
transform_extra_arguments=(("metadata", "metadata"),),
fit_tag="q_short_gpu",
)
from bob.pipelines.distributed.sge import get_resource_requirements
resources = get_resource_requirements(pipeline)
pipeline.fit_transform(X_as_sample).compute(
scheduler=client, resources=resources
)
```https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/53Updates2020-11-26T15:13:57ZTiago de Freitas PereiraUpdatesImplemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Implemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/52[CheckpointWrapper] Allow custom save and load functions through estimator tags2020-11-27T11:17:16ZAmir MOHAMMADI[CheckpointWrapper] Allow custom save and load functions through estimator tagsBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/51[dask][sge] Multiqueue updates2020-11-25T16:51:11ZTiago de Freitas Pereira[dask][sge] Multiqueue updatesIn this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "chec...In this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path=model_path,
fit_tag="q_short_gpu",
)
```
You have to explicitly set the list of resource tags available.
```python
pipeline.fit_transform(...).compute(
scheduler=dask_client, resources=cluster.get_sge_resources()
```Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/50[DelayedSample(Set)] make load and delayed_attributes private2020-11-23T16:24:44ZAmir MOHAMMADI[DelayedSample(Set)] make load and delayed_attributes privateThis removes the need for a lot of guessing in downstream packages as they can start removing all keys that start with `_` when access of the sample's attribute is needed.This removes the need for a lot of guessing in downstream packages as they can start removing all keys that start with `_` when access of the sample's attribute is needed.Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/49[DelayedSample] Fix issues when an attribute was set2020-11-23T14:19:57ZAmir MOHAMMADI[DelayedSample] Fix issues when an attribute was setBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/48Improvements on CheckpointWrapper2020-11-22T18:40:31ZTiago de Freitas PereiraImprovements on CheckpointWrapperAdded the optional argument `hash_fn` in the `CheckpointWrapper` class.
Once this is set, `sample.key` generates a hash code and this hash code is used to compose the final path where `sample` will be checkpointed.
This is optional and ...Added the optional argument `hash_fn` in the `CheckpointWrapper` class.
Once this is set, `sample.key` generates a hash code and this hash code is used to compose the final path where `sample` will be checkpointed.
This is optional and generic enough for our purposes.
This hash function can be shipped in the database interface.
Closes https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/25Bob 9.0.0Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/47Multiple changes2020-11-23T10:27:21ZAmir MOHAMMADIMultiple changes* [DelayedSample] Allow for arbitrary delayed attributes
* [SampleBatch] Allow other attributes than data
Fixes #26 #24* [DelayedSample] Allow for arbitrary delayed attributes
* [SampleBatch] Allow other attributes than data
Fixes #26 #24Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/46move vstack_features to bob.io.base2020-11-13T11:02:21ZAmir MOHAMMADImove vstack_features to bob.io.baseBob 9.0.0Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/45Multiple Changes2020-11-11T08:17:48ZAmir MOHAMMADIMultiple Changes* When checkpointing, checkpoing all steps in a pipeline
* Better names in dask graph for FunctionTransformer
* [xarray] Allow for multi argument transformers
* SampleBatch in public API* When checkpointing, checkpoing all steps in a pipeline
* Better names in dask graph for FunctionTransformer
* [xarray] Allow for multi argument transformers
* SampleBatch in public APIBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/44Fix sphinx warnings2020-11-09T14:48:34ZAmir MOHAMMADIFix sphinx warningsConda-based CIAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/43Allow setting specific attributes of sample2020-11-06T15:13:29ZYannick DAYERAllow setting specific attributes of sampleSpecify the sample attribute to assign the output of an estimator to, instead of 'data' in SampleWrapper.
Specify the attribute of sample to save and load in CheckpointWrapper.Specify the sample attribute to assign the output of an estimator to, instead of 'data' in SampleWrapper.
Specify the attribute of sample to save and load in CheckpointWrapper.Bob 9.0.0Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/42Adding annotations-related wrappers2020-11-06T14:38:16ZYannick DAYERAdding annotations-related wrappersAdds an annotated Sample wrapper that wraps an annotator, feeding it `sample.data` and storing the resulting annotations in `sample.annotations`.
Also adds an annotations Checkpoint wrapper that saves and loads a `sample.annotations` to...Adds an annotated Sample wrapper that wraps an annotator, feeding it `sample.data` and storing the resulting annotations in `sample.annotations`.
Also adds an annotations Checkpoint wrapper that saves and loads a `sample.annotations` to the disk.Bob 9.0.0Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/41Added a GPU queue that defaults to short_gpu2020-11-05T15:22:13ZYannick DAYERAdded a GPU queue that defaults to short_gpuBob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/40Add dask-client configurations as resources2020-10-12T14:19:51ZYannick DAYERAdd dask-client configurations as resourcesFixes #19
Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.
Depends on bob.bio.base!201Fixes #19
Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.
Depends on bob.bio.base!201Bob 9.0.0Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/39[dask][sge] Added the variables `idle_timeout` and `allowed_failures` as2020-10-12T15:04:50ZTiago de Freitas Pereira[dask][sge] Added the variables `idle_timeout` and `allowed_failures` aspart of our `.bobrc` and added better defaultspart of our `.bobrc` and added better defaultsBob 9.0.0Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/38[sge] In dask some sublacessd classes need a config name. Fixes #202020-10-12T11:25:49ZAmir MOHAMMADI[sge] In dask some sublacessd classes need a config name. Fixes #20Bob 9.0.0Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/37Revert "For some reason, the class information is not passed in the sample wr...2020-10-06T16:43:02ZAmir MOHAMMADIRevert "For some reason, the class information is not passed in the sample wrapper"This reverts merge request !36This reverts merge request !36Tiago de Freitas PereiraTiago de Freitas Pereira