bob.pipelines merge requestshttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests2021-10-29T15:34:58Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/69[SampleSet] Do not load delayed attributes by not copying them over2021-10-29T15:34:58ZAmir MOHAMMADI[SampleSet] Do not load delayed attributes by not copying them overAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/71[CheckpointWrapper] Use atomic writing when saving features2021-10-29T15:34:58ZAmir MOHAMMADI[CheckpointWrapper] Use atomic writing when saving featuresTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/72+ breakdown_SampleSet2021-10-29T15:34:58ZHatef OTROSHI+ breakdown_SampleSetAs desciribed in https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/33,
a function is added which takes as input a `SampleSet` with `N` samples and outputs `N` `SampleSets` with 1 `Sample` each.
ping @tiago.pereira
Closes #33 As desciribed in https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/33,
a function is added which takes as input a `SampleSet` with `N` samples and outputs `N` `SampleSets` with 1 `Sample` each.
ping @tiago.pereira
Closes #33 https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/73Remove samples_to_hdf5 methods2021-10-29T15:34:58ZAmir MOHAMMADIRemove samples_to_hdf5 methodsThese methods were not used anywhere.These methods were not used anywhere.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/74Add a DelayedSample.from_sample classmethod2021-10-29T15:34:58ZAmir MOHAMMADIAdd a DelayedSample.from_sample classmethodThis method can be used to transparently create new
DelayedSamples from either Samples or DelayedSamples without loading
delayed attributes and dataThis method can be used to transparently create new
DelayedSamples from either Samples or DelayedSamples without loading
delayed attributes and dataTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/75Add worker Time To Live limitation2021-10-29T15:34:58ZLaurent COLBOISAdd worker Time To Live limitationHello,
I have regularly been annoyed by Dask runs that hang indefinitely because of some workers being disconnected from the scheduler. In this case, the scheduler actually assumes the worker must still be doing its job so it doesn't rea...Hello,
I have regularly been annoyed by Dask runs that hang indefinitely because of some workers being disconnected from the scheduler. In this case, the scheduler actually assumes the worker must still be doing its job so it doesn't reassign the task, leading to a completely blocked run that needs to be interrupted by hand. This typically happens on very heavy experiments e.g. on IJBC, FRGC.
From what I understand this can be handled using the `worker_ttl` parameter of the scheduler, which puts a limit on how long a worker can be unseen by the scheduler before being killed and reassigning its task. It is `None` by default, I have been working for a while on a local branch where I set the default to 60s, it helped quite a lot.
I am proposing to merge this change, however I wanted to know what you think of it. My main concern is that it might be hiding some underlying issue (why do the workers actually disconnect ?), so I am not 100% sure it's a good change to make.
ping @tiago.pereira @amohammadihttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/76[docs] update docs to match new API of xarray2021-10-29T15:34:58ZAmir MOHAMMADI[docs] update docs to match new API of xarrayFixes #35
Disabled testing Sphinx docs on mac builds.Fixes #35
Disabled testing Sphinx docs on mac builds.Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/78pin dask versions more strictly2021-12-06T10:06:34ZAmir MOHAMMADIpin dask versions more strictlyFixes #40Fixes #40Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/79Fixing compatibility issues with dask_jobqueue=0.7.22021-11-30T18:25:54ZTiago de Freitas PereiraFixing compatibility issues with dask_jobqueue=0.7.2closes #37
Unfortunately we can't test this on the CI (there's no SGE there)closes #37
Unfortunately we can't test this on the CI (there's no SGE there)https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/77Resolve "local-parallel queue is not setup well"2021-12-06T11:09:29ZManuel Günthersiebenkopf@googlemail.comResolve "local-parallel queue is not setup well"Closes #38Closes #38Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/80Implemented a force mechanism2022-01-13T13:09:57ZTiago de Freitas PereiraImplemented a force mechanismCreated a `force` option for the CheckpointWrapper
Related to:
https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/173Created a `force` option for the CheckpointWrapper
Related to:
https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/173Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/70Handle estimator tags in wrapper classes2022-03-18T09:09:58ZYannick DAYERHandle estimator tags in wrapper classesAllows setting some parameters of the `SampleWrapper` and `CheckpointWrapper` via estimator tags.
bob.bio.base#143Allows setting some parameters of the `SampleWrapper` and `CheckpointWrapper` via estimator tags.
bob.bio.base#143Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/81Fix get_bob_tags to return default tags2022-02-21T13:11:13ZYannick DAYERFix get_bob_tags to return default tagsWhen passing `None` as estimator to get_bob_tags, returns the default tags.When passing `None` as estimator to get_bob_tags, returns the default tags.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/82breaking: checkpoint the inner estimator only2022-03-31T14:14:43ZAmir MOHAMMADIbreaking: checkpoint the inner estimator onlyAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/84Created a function checking if a Scikit learn pipeline is wrapped2022-03-16T17:37:34ZTiago de Freitas PereiraCreated a function checking if a Scikit learn pipeline is wrappedCreated the function `is_estimator_wrapped`Created the function `is_estimator_wrapped`https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/83[dask] Convert dask bags to arrays more efficiently2022-03-18T11:48:04ZAmir MOHAMMADI[dask] Convert dask bags to arrays more efficientlyMost inefficiencies were coming from that fact that we were creating a dask array with each sample as a separate chunk.Most inefficiencies were coming from that fact that we were creating a dask array with each sample as a separate chunk.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/85Prevent a reference invalidation when wrapped with sample and checkpoint.2022-04-05T09:41:52ZYannick DAYERPrevent a reference invalidation when wrapped with sample and checkpoint.Prevents creating a new estimator when loading a sample-wrapped estimator with `CheckpointWrapper` (continuation of !82 which prevented the creation of a new estimator right "below").
This now checks if the estimator is wrapped with `Sa...Prevents creating a new estimator when loading a sample-wrapped estimator with `CheckpointWrapper` (continuation of !82 which prevented the creation of a new estimator right "below").
This now checks if the estimator is wrapped with `SampleWrapper` and updates the estimator at that level.
Fixes bob.bio.gmm#30.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/86Fix fit extra parameters2022-04-25T17:44:22ZYannick DAYERFix fit extra parametersAllowed extra fit parameters to be non-array (e.g. str).
Added a tag to prevent stacking of the input array of the fit method if it expects partitioned data.Allowed extra fit parameters to be non-array (e.g. str).
Added a tag to prevent stacking of the input array of the fit method if it expects partitioned data.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/87Add a non-adaptive io-big queue2022-04-27T19:14:00ZAmir MOHAMMADIAdd a non-adaptive io-big queueAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/88Add support for fitting estimators on dask bags2022-05-09T14:31:58ZAmir MOHAMMADIAdd support for fitting estimators on dask bagsThe estimators that can handle dask bags should set
the `bob_fit_supports_dask_bag` as True.
This commit also includes
* Adds a new tag: `bob_fit_supports_dask_bag`
* Adds a new tag: `bob_checkpoint_features` for when you want to always ...The estimators that can handle dask bags should set
the `bob_fit_supports_dask_bag` as True.
This commit also includes
* Adds a new tag: `bob_fit_supports_dask_bag`
* Adds a new tag: `bob_checkpoint_features` for when you want to always avoid
checkpointing features for a specific estimator.
* Expose dask_tags, get_bob_tags in the main API
* The SampleWrapper was modified to support `bob_fit_supports_dask_bag`
* The CheckpointWrapper now loads estimators without losing references correctly.The Great DeprecationAmir MOHAMMADIAmir MOHAMMADI