bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2022-06-28T14:00:34Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/96Fix Dask documentation2022-06-28T14:00:34ZFlavio TARSETTIFix Dask documentationThis MR fixes the issues with the Dask documentationThis MR fixes the issues with the Dask documentationFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/95[datasets] remove fieldnames from CSVToSamples2022-06-13T10:11:25ZAmir MOHAMMADI[datasets] remove fieldnames from CSVToSamplesSince it can be provided as part of dict_reader_kwargs
Also reorganize the DelayedSample code and tests
to make things more readableSince it can be provided as part of dict_reader_kwargs
Also reorganize the DelayedSample code and tests
to make things more readableThe Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/94[hotfix] [Dask] Add configuration to access multithreaded queue2022-06-14T15:15:30ZLaurent COLBOIS[hotfix] [Dask] Add configuration to access multithreaded queueThis adds an additional queue configuration that asks for multithreaded jobs, thus doubling the virtual memory limit to 16GB.
With this fix I can run heavy Tensorflow baselines without OOM crash.
Long term it would be better to:
* Expos...This adds an additional queue configuration that asks for multithreaded jobs, thus doubling the virtual memory limit to 16GB.
With this fix I can run heavy Tensorflow baselines without OOM crash.
Long term it would be better to:
* Expose the exact thread multiplier (now 2 by default)
* Use the resource tagging of transformers to have the queue dynamically selected based on the transformerhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/93Add documentation for CSV databases2022-06-02T14:16:56ZAmir MOHAMMADIAdd documentation for CSV databasesThe Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/92better logging overall2022-05-20T12:15:42ZAmir MOHAMMADIbetter logging overallThe Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/91Many API changes2022-05-24T11:40:31ZAmir MOHAMMADIMany API changesExpose utils API in the root API.
Fix the docs API.
Remove unused transformers.
Fix SGE GPU submissions.Expose utils API in the root API.
Fix the docs API.
Remove unused transformers.
Fix SGE GPU submissions.The Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/90replace is_estimator_stateless with estimator_requires_fit2022-05-09T18:30:39ZAmir MOHAMMADIreplace is_estimator_stateless with estimator_requires_fitThe actual code before meant to check if an estimator
requires fit or not while the function was named is_estimator_stateless.The actual code before meant to check if an estimator
requires fit or not while the function was named is_estimator_stateless.The Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/89Load checkpointed estimators inside the scheduler2022-05-09T17:40:41ZAmir MOHAMMADILoad checkpointed estimators inside the schedulerAlso adds resilience to loading checkpointed samplesAlso adds resilience to loading checkpointed samplesThe Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/88Add support for fitting estimators on dask bags2022-05-09T14:31:58ZAmir MOHAMMADIAdd support for fitting estimators on dask bagsThe estimators that can handle dask bags should set
the `bob_fit_supports_dask_bag` as True.
This commit also includes
* Adds a new tag: `bob_fit_supports_dask_bag`
* Adds a new tag: `bob_checkpoint_features` for when you want to always ...The estimators that can handle dask bags should set
the `bob_fit_supports_dask_bag` as True.
This commit also includes
* Adds a new tag: `bob_fit_supports_dask_bag`
* Adds a new tag: `bob_checkpoint_features` for when you want to always avoid
checkpointing features for a specific estimator.
* Expose dask_tags, get_bob_tags in the main API
* The SampleWrapper was modified to support `bob_fit_supports_dask_bag`
* The CheckpointWrapper now loads estimators without losing references correctly.The Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/87Add a non-adaptive io-big queue2022-04-27T19:14:00ZAmir MOHAMMADIAdd a non-adaptive io-big queueAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/86Fix fit extra parameters2022-04-25T17:44:22ZYannick DAYERFix fit extra parametersAllowed extra fit parameters to be non-array (e.g. str).
Added a tag to prevent stacking of the input array of the fit method if it expects partitioned data.Allowed extra fit parameters to be non-array (e.g. str).
Added a tag to prevent stacking of the input array of the fit method if it expects partitioned data.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/85Prevent a reference invalidation when wrapped with sample and checkpoint.2022-04-05T09:41:52ZYannick DAYERPrevent a reference invalidation when wrapped with sample and checkpoint.Prevents creating a new estimator when loading a sample-wrapped estimator with `CheckpointWrapper` (continuation of !82 which prevented the creation of a new estimator right "below").
This now checks if the estimator is wrapped with `Sa...Prevents creating a new estimator when loading a sample-wrapped estimator with `CheckpointWrapper` (continuation of !82 which prevented the creation of a new estimator right "below").
This now checks if the estimator is wrapped with `SampleWrapper` and updates the estimator at that level.
Fixes bob.bio.gmm#30.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/84Created a function checking if a Scikit learn pipeline is wrapped2022-03-16T17:37:34ZTiago de Freitas PereiraCreated a function checking if a Scikit learn pipeline is wrappedCreated the function `is_estimator_wrapped`Created the function `is_estimator_wrapped`https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/83[dask] Convert dask bags to arrays more efficiently2022-03-18T11:48:04ZAmir MOHAMMADI[dask] Convert dask bags to arrays more efficientlyMost inefficiencies were coming from that fact that we were creating a dask array with each sample as a separate chunk.Most inefficiencies were coming from that fact that we were creating a dask array with each sample as a separate chunk.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/82breaking: checkpoint the inner estimator only2022-03-31T14:14:43ZAmir MOHAMMADIbreaking: checkpoint the inner estimator onlyAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/81Fix get_bob_tags to return default tags2022-02-21T13:11:13ZYannick DAYERFix get_bob_tags to return default tagsWhen passing `None` as estimator to get_bob_tags, returns the default tags.When passing `None` as estimator to get_bob_tags, returns the default tags.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/80Implemented a force mechanism2022-01-13T13:09:57ZTiago de Freitas PereiraImplemented a force mechanismCreated a `force` option for the CheckpointWrapper
Related to:
https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/173Created a `force` option for the CheckpointWrapper
Related to:
https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/173Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/79Fixing compatibility issues with dask_jobqueue=0.7.22021-11-30T18:25:54ZTiago de Freitas PereiraFixing compatibility issues with dask_jobqueue=0.7.2closes #37
Unfortunately we can't test this on the CI (there's no SGE there)closes #37
Unfortunately we can't test this on the CI (there's no SGE there)https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/78pin dask versions more strictly2021-12-06T10:06:34ZAmir MOHAMMADIpin dask versions more strictlyFixes #40Fixes #40Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/77Resolve "local-parallel queue is not setup well"2021-12-06T11:09:29ZManuel Günthersiebenkopf@googlemail.comResolve "local-parallel queue is not setup well"Closes #38Closes #38Amir MOHAMMADIAmir MOHAMMADI