bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2021-10-29T15:34:58Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/69[SampleSet] Do not load delayed attributes by not copying them over2021-10-29T15:34:58ZAmir MOHAMMADI[SampleSet] Do not load delayed attributes by not copying them overAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/68Fix parent's delayed_attributes modified by child2021-10-29T15:34:58ZYannick DAYERFix parent's delayed_attributes modified by childA `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.A `DelayedSample` child's `delayed_attributes` is no longer referencing the parent's `delayed_attributes`.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/67Some minor updates on the checkpoint wrapper and SGE2021-10-29T15:34:58ZTiago de Freitas PereiraSome minor updates on the checkpoint wrapper and SGEYannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/66Handled failed processing (Failure to Acquire) in the wrappers2021-10-29T15:34:58ZYannick DAYERHandled failed processing (Failure to Acquire) in the wrappersFixes #32Fixes #32Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/64Fix delayed attributes2021-10-29T15:34:58ZYannick DAYERFix delayed attributesDelayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Delayed attributes are no longer loaded when `_copy_attributes` is called to create a DelayedSample.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/63Implemented a mechanism in the Checkpoint wrapper that asserts if data was...2021-10-29T15:34:58ZTiago de Freitas PereiraImplemented a mechanism in the Checkpoint wrapper that asserts if data was...Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Implemented a mechanism in the Checkpoint wrapper that asserts if data was properly written in the disk
Closes #31Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/61Fixed modules2021-02-02T10:44:41ZTiago de Freitas PereiraFixed modulesconfig files from here are not available once `conda install bob.pipelines`config files from here are not available once `conda install bob.pipelines`https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60CSVSampleLoaders as transformers2020-12-14T13:59:48ZTiago de Freitas PereiraCSVSampleLoaders as transformersMade CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read ann...Made CSVSampleLoaders as scikit-learn transformers
This is a good idea indeed.
I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read annotations using `bob.db.base.read_anno...`.
This is delayed.
I'm already porting this stuff on `bob.bio.base`. Code is way more cleaner.
ping @amohammadi @ydayer
Closes #30 Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/58Moved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a g...2020-12-08T16:24:03ZTiago de Freitas PereiraMoved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a general functionhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/56Two new features2020-12-07T13:08:49ZTiago de Freitas PereiraTwo new features- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-up- Moved dask_get_partition_size from bob.bio.base to bob.pipelines
- Updated the target duration of a task to 10s. Being very aggressive in scale-uphttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/55Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines2020-12-09T10:15:36ZTiago de Freitas PereiraMoved VALID_DASK_CLIENT_STRINGS to bob.pipelineshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/54Fixed multiqueue2020-11-27T15:34:52ZTiago de Freitas PereiraFixed multiqueueHi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in t...Hi @amohammadi @ydayer
I'm fixing here the issue raised with the multiqueue.
I was wrongly setting all tasks to run in a particular resource restriction.
Now the problem is fixed.
To get it running you have to wrap your pipeline in the same
way as before and fetch the resources like this
```python
pipeline = bob.pipelines.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path="./",
transform_extra_arguments=(("metadata", "metadata"),),
fit_tag="q_short_gpu",
)
from bob.pipelines.distributed.sge import get_resource_requirements
resources = get_resource_requirements(pipeline)
pipeline.fit_transform(X_as_sample).compute(
scheduler=client, resources=resources
)
```https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/53Updates2020-11-26T15:13:57ZTiago de Freitas PereiraUpdatesImplemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Implemented two updates in this MR
- Removed the random behavior on the hash_string function (i had some problems in large scale tests).
- Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/51[dask][sge] Multiqueue updates2020-11-25T16:51:11ZTiago de Freitas Pereira[dask][sge] Multiqueue updatesIn this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "chec...In this merge request I:
- Simplified the way multi-queue is set in our scripts
- Updated our Dask documentation
Example
-------
Setting the `fit` method to run on `q_short_gpu`
```python
pipeline = mario.wrap(
["sample", "checkpoint", "dask"],
pipeline,
model_path=model_path,
fit_tag="q_short_gpu",
)
```
You have to explicitly set the list of resource tags available.
```python
pipeline.fit_transform(...).compute(
scheduler=dask_client, resources=cluster.get_sge_resources()
```Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/37Revert "For some reason, the class information is not passed in the sample wr...2020-10-06T16:43:02ZAmir MOHAMMADIRevert "For some reason, the class information is not passed in the sample wrapper"This reverts merge request !36This reverts merge request !36Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/36For some reason, the class information is not passed in the sample wrapper2020-08-31T10:13:47ZTiago de Freitas PereiraFor some reason, the class information is not passed in the sample wrapperhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/35Add DelayedSampleSet, remove bob.pipelines script, add pre-commit2020-07-24T12:23:40ZAmir MOHAMMADIAdd DelayedSampleSet, remove bob.pipelines script, add pre-commitTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/34Project variable2020-07-24T10:59:20ZTiago de Freitas PereiraProject variableAdded `-P <project>` to the SGEClusterAdded `-P <project>` to the SGEClusterTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/33SGE defaults2020-07-07T11:50:31ZTiago de Freitas PereiraSGE defaultsMost of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.Most of the modifications are changing scheduler defaults but temporally added `biometric` as a default for the SGE `-P` option.
I'll merge it because I need this quickly to be on the master for this afternoon.https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/32Created a method that emulates isinsntace for aggregation2020-06-18T12:36:15ZTiago de Freitas PereiraCreated a method that emulates isinsntace for aggregationThis will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something elseThis will allow us to check if something wrapped with dask is "checkpointable", or "samplable" or something else