bob issueshttps://gitlab.idiap.ch/groups/bob/-/issues2022-05-31T09:21:15Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/68bob.measure.plot.det() yields points out of bounds in some cases2022-05-31T09:21:15ZAlex UNNERVIKbob.measure.plot.det() yields points out of bounds in some casesIt seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
...It seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
```
import bob.measure
import matplotlib.pyplot as plt
import numpy as np
plt.figure()
l1 = bob.measure.plot.det([1, 2, 3], [2, 2.5, 4], npoints=200) # works for apparently any number of points, including 3 and 2000
print(np.max((l1[0].get_ydata()))) # yields 8.126357928110227
print(np.min((l1[0].get_ydata()))) # yields -8.126357928110227
print(np.max((l1[0].get_xdata()))) # yields 8.126357928110227
print(np.min((l1[0].get_xdata()))) # yields -8.126357928110227
```
These values are inconsistent with expected values in the range of [0,1].
The image shows the result of the plot of the above code, simply adding `plt.show()`.![bob_det_plot](/uploads/9f7eb0264fac5629eeee1d4c4c633c6a/bob_det_plot.png)https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/185Probe template not accepted if not numpy array2022-05-31T08:57:06ZYannick DAYERProbe template not accepted if not numpy arrayIn `score_sample_template` from `abstract_classes.py` there is a check that the data is "valid" ([here](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/master/bob/bio/base/pipelines/abstract_classes.py#L246)).
It checks that the templat...In `score_sample_template` from `abstract_classes.py` there is a check that the data is "valid" ([here](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/master/bob/bio/base/pipelines/abstract_classes.py#L246)).
It checks that the template in the samples is a Numpy array and is not empty. But in GMM for example, the templates are objects (`GMMStats`) and all the samples are considered "invalid" and ignored silently.
We could either remove the check or implement a way of detecting non-Numpy objects and allow those unconditionally.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.bio.spear/-/issues/42ImportError: cannot import name 'SpeechbrainEmbeddings' from 'bob.bio.spear.e...2022-05-25T15:00:39ZHatef OTROSHIImportError: cannot import name 'SpeechbrainEmbeddings' from 'bob.bio.spear.extractor'In the master branch, in [bob/bio/spear/config/pipeline/speechbrain_ecapa_voxceleb.py#L4](https://gitlab.idiap.ch/bob/bob.bio.spear/-/blob/ddf128c6da2872d7ec5a2608b4fddf03b3884537/bob/bio/spear/config/pipeline/speechbrain_ecapa_voxceleb....In the master branch, in [bob/bio/spear/config/pipeline/speechbrain_ecapa_voxceleb.py#L4](https://gitlab.idiap.ch/bob/bob.bio.spear/-/blob/ddf128c6da2872d7ec5a2608b4fddf03b3884537/bob/bio/spear/config/pipeline/speechbrain_ecapa_voxceleb.py#L4) there is `from bob.bio.spear.extractor import SpeechbrainEmbeddings` but `SpeechbrainEmbeddings` cannot be imported.https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/184cannot import 'isinstance_nested' from 'bob.pipelines.utils'2022-05-24T17:23:17ZHatef OTROSHIcannot import 'isinstance_nested' from 'bob.pipelines.utils'In the master branch, in [bob/bio/base/pipelines/pipelines.py#L18](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/1abaa5f11559c15ee185203a65f8480800524b01/bob/bio/base/pipelines/pipelines.py#L18) there is `from bob.pipelines.utils impor...In the master branch, in [bob/bio/base/pipelines/pipelines.py#L18](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/1abaa5f11559c15ee185203a65f8480800524b01/bob/bio/base/pipelines/pipelines.py#L18) there is `from bob.pipelines.utils import isinstance_nested` but `isinstance_nested` is not available in master branch of [bob/pipelines/utils.py](https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/611f8867a16711bf0394cc9af2901bd4941b6a5b/bob/pipelines/utils.py)https://gitlab.idiap.ch/bob/bob.pad.face/-/issues/48VideoPadSample's output is not consistent2022-06-09T12:54:04ZChristophe ECABERTVideoPadSample's output is not consistentThe samples generated by the `bob.pad.face.database.VideoPadSample` are not consistent. The attributes `.data` and `.annotations` do not have the same first dimensions. This behaviour can be reproduce with:
```python
database = ReplayAt...The samples generated by the `bob.pad.face.database.VideoPadSample` are not consistent. The attributes `.data` and `.annotations` do not have the same first dimensions. This behaviour can be reproduce with:
```python
database = ReplayAttackPadDatabase(protocol='smalltest')
samples = database.fit_samples()
vid0 = samples[0].data # vid0.shape == (20, 3, 240, 320)
ann0 = samples[0].annotations # len(ann0) == 375
```
The `.data` attribute contains the sampled frames (i.e. by `VideoAsArray` container) whereas the `.annotations` attribute contains the annotations for the whole video (i.e. they are not sampled)https://gitlab.idiap.ch/bob/bob.pad.face/-/issues/47Vanilla-pad baseline does not run2022-05-31T08:57:22ZChristophe ECABERTVanilla-pad baseline does not runFollowing the [documentation](https://www.idiap.ch/software/bob/docs/bob/docs/master/bob/bob.pad.base/doc/vanilla_pad_intro.html#running-a-biometric-experiment-with-vanilla-pad), the vanilla-pad baseline can be invoked with:
```python
b...Following the [documentation](https://www.idiap.ch/software/bob/docs/bob/docs/master/bob/bob.pad.base/doc/vanilla_pad_intro.html#running-a-biometric-experiment-with-vanilla-pad), the vanilla-pad baseline can be invoked with:
```python
bob pad vanilla-pad replay-attack svm-frames -o results -vv
```
This leads to the following exception:
```python
Traceback (most recent call last):
File "/remote/idiap.svm/temp.biometric03/cecabert/bob_beta/src/bob.pipelines/bob/pipelines/wrappers.py", line 808, in _fit
self.estimator = self.estimator.fit(X, y, **fit_params)
File "/remote/idiap.svm/temp.biometric03/cecabert/bob_beta/src/bob.pipelines/bob/pipelines/wrappers.py", line 337, in fit
self.estimator = self.estimator.fit(X, **kwargs)
File "/remote/idiap.svm/temp.biometric03/cecabert/mambaforge/envs/bob_deps/lib/python3.10/site-packages/sklearn/svm/_base.py", line 190, in fit
X, y = self._validate_data(
File "/remote/idiap.svm/temp.biometric03/cecabert/mambaforge/envs/bob_deps/lib/python3.10/site-packages/sklearn/base.py", line 581, in _validate_data
X, y = check_X_y(X, y, **check_params)
File "/remote/idiap.svm/temp.biometric03/cecabert/mambaforge/envs/bob_deps/lib/python3.10/site-packages/sklearn/utils/validation.py", line 964, in check_X_y
X = check_array(
File "/remote/idiap.svm/temp.biometric03/cecabert/mambaforge/envs/bob_deps/lib/python3.10/site-packages/sklearn/utils/validation.py", line 794, in check_array
raise ValueError(
ValueError: Found array with dim 4. Estimator expected <= 2.
```
The `fit` method of the SVM classifier is expecting its input to be `(n_samples, n_features)` which is not what is provided by the `VideoToFrames` transformer. One possible solution would be to add an intermediate step between the two operations to flatten the data.https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/41Expanding samples on the fly2022-05-31T13:53:53ZAnjith GEORGEanjith.george@idiap.chExpanding samples on the flyI have a usecase where I want to expand one image to multiple images in the `pipelinesimple`. Essentially, I can create
a transformer which can operate directly on samples, which takes in a `SampleSet` with one image and return a `Sampl...I have a usecase where I want to expand one image to multiple images in the `pipelinesimple`. Essentially, I can create
a transformer which can operate directly on samples, which takes in a `SampleSet` with one image and return a `SampleSet` with `n` samples.
I managed to make it work, when every thing is in memory (with `-m` option), and checkpointing is a problem since there are new samples. I can create new keys on the fly, what do you think is the best way to go about this.
@amohammadi @tiago.pereira @ydayerhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/183Heterogeneous pipelines for identification2022-05-31T13:54:33ZAnjith GEORGEanjith.george@idiap.chHeterogeneous pipelines for identificationI am trying to run a face recognition pipline where the pipeline for enrollment and probes are different, basically my enrollment part has one step extra. Currently I have created a new heterogeneous pipeline which takes in two different...I am trying to run a face recognition pipline where the pipeline for enrollment and probes are different, basically my enrollment part has one step extra. Currently I have created a new heterogeneous pipeline which takes in two different pipelines as the inputs. I think its not very efficient, since there are common components in both of the pipelines (Iresnet in my case). I think it would be cleaner, if there is a transform which can do things conditionally, meaning, if it sees that the sample is from the enrollment set then apply the transform and skup it if its not from enrollment. Adding the group attribute at the csvloader level might solve this, so that I can do the filtering in this transform.
Any suggestions welcome.https://gitlab.idiap.ch/bob/bob.bio.spear/-/issues/41Memory issues with big dataset2022-05-16T15:02:59ZYannick DAYERMemory issues with big datasetWhen running a GMM experiment with a big dataset (Nist-SRE04to16) the pipeline fails with a memory error on a worker.
for example on the branch of !55:
```bob bio pipeline simple -vvv -d nist-sre04to16 -p gmm-voxforge -o results\~/gmm_...When running a GMM experiment with a big dataset (Nist-SRE04to16) the pipeline fails with a memory error on a worker.
for example on the branch of !55:
```bob bio pipeline simple -vvv -d nist-sre04to16 -p gmm-voxforge -o results\~/gmm_nist -l sge```
I ran with the default Dask `sge` client as well as the `sge-io-big-non-adaptive`, asking for 128 nodes (but got only ~60 while running).
The issue seems to happen before reaching the k-means initialization, maybe hinting at an issue in the Dask bags to array wrapping.
I also tried running the experiment with a lower Dask memory limit for each node, forcing the workers to spill their memory to disk early, trying to prevent the memory error if it reached the hard cap. This failed too (the workers effectively spilled to disk but still failed with a memory error).
<details><summary>Local Output and Traceback (Click to expand)</summary>
```
[...]
bob.pipelines.wrappers@2022-05-16 13:04:59,280 -- DEBUG: ToDaskBag(npartitions=128).transform
bob.pipelines.wrappers@2022-05-16 13:04:59,926 -- DEBUG: Dask|Checkpoint|Sample|Energy_.transform
bob.pipelines.wrappers@2022-05-16 13:04:59,927 -- DEBUG: Dask|Checkpoint|Sample|Cepstra.transform
bob.pipelines.wrappers@2022-05-16 13:04:59,929 -- DEBUG: Dask|Checkpoint|Sample|GMM(con.fit
bob.pipelines.wrappers@2022-05-16 13:04:59,941 -- DEBUG: Preparing data as dask arrays for fit
Traceback (most recent call last):
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/bin/bob", line 10, in <module>
sys.exit(main_cli())
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.bio.base/bob/bio/base/script/pipeline_simple.py", line 276, in pipeline_simple
execute_pipeline_simple(
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.bio.base/bob/bio/base/pipelines/entry_points.py", line 225, in execute_pipeline_simple
result = pipeline(
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.bio.base/bob/bio/base/pipelines/pipelines.py", line 109, in __call__
self.transformer = self.train_background_model(background_model_samples)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.bio.base/bob/bio/base/pipelines/pipelines.py", line 144, in train_background_model
return self.transformer.fit(background_model_samples)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/sklearn/pipeline.py", line 394, in fit
self._final_estimator.fit(Xt, y, **fit_params_last_step)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 881, in fit
return self._fit_on_dask_array(X, y, **fit_params)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 835, in _fit_on_dask_array
X, fit_params = self._get_fit_params_from_sample_bags(bags)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 816, in _get_fit_params_from_sample_bags
X = _array_from_sample_bags(bags, input_attribute, ndim=2)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 693, in _array_from_sample_bags
lengths, shapes = dask.compute(lengths, shapes)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/dask/base.py", line 573, in compute
results = schedule(dsk, keys, **kwargs)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/client.py", line 3010, in get
results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/client.py", line 2162, in gather
return self.sync(
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/utils.py", line 311, in sync
return sync(
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/utils.py", line 378, in sync
raise exc.with_traceback(tb)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/utils.py", line 351, in f
result = yield future
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/distributed/client.py", line 2025, in _gather
raise exception.with_traceback(traceback)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/dask/utils.py", line 39, in apply
return func(*args, **kwargs)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 664, in _sample_attribute
return [getattr(s, attribute) for s in samples]
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/wrappers.py", line 664, in <listcomp>
return [getattr(s, attribute) for s in samples]
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/sample.py", line 170, in __getattribute__
return super().__getattribute__(name)
File "/remote/idiap.svm/temp.devel01/ydayer/spear_develop/bob.pipelines/bob/pipelines/sample.py", line 188, in data
return self._load()
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/bob/io/base/__init__.py", line 191, in load
return open_file(inputs)
File "/idiap/home/ydayer/miniconda3/envs/spear_bob10/lib/python3.8/site-packages/bob/io/base/__init__.py", line 101, in open_file
return np.array(f[key])
numpy.core._exceptions.MemoryError: Unable to allocate 7.89 MiB for an array with shape (17226, 60) and data type float64
```
</details>https://gitlab.idiap.ch/bob/bob/-/issues/273Issues to take care of before release2022-06-10T11:33:30ZAmir MOHAMMADIIssues to take care of before release- [ ] at the end of many pages e.g. https://www.idiap.ch/software/bob/docs/bob/docs/master/bob/bob.bio.spear/README.html there is a contact section with "For questions or reporting issues to this software package, contact our development...- [ ] at the end of many pages e.g. https://www.idiap.ch/software/bob/docs/bob/docs/master/bob/bob.bio.spear/README.html there is a contact section with "For questions or reporting issues to this software package, contact our development mailing list." pointing to https://groups.google.com/g/bob-devel -- isn't it a bit misleading since we are asking people to ask questions on stackoverflow https://stackoverflow.com/questions/tagged/python-bob ?
- [ ] bob.extension is still called a C++ extension package and the documentation needs to be revisited. Most of the useful docs for development should move to bob/bob.devtools
- [x] The tutorial in bob/docs is gone and we need a new one. The other pages need to be revised.
- [ ] bob/bob.bio.base documentation on creating csv datasets is not complete
- [ ] bob/bob.bio.base documentation on legacy should be removed
- [ ] bob/bob.bio.spear documentation and leaderboard needs updating
- [ ] bob/bob.bio.face leaderboard needs updating
- [ ] bob/bob.pad.face needs new baselines. the package is empty as of now.
- [ ] bob/bob.pad.face needs new documentation after baselines have changed.
- [x] bob/bob.bio.vein still uses nosetests instead of pytest
- [ ] bob/bob.devtools we need to command to install all packages inside `bob-devel`. That is creating a conda env with all bob dependencies and without bob packages.
- [x] Check the entry points of all packages (check which ones).The Great Deprecationhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/182Unable to run the pipeline after the last update of bob.bio.base2022-05-03T16:26:00ZYu LinghuUnable to run the pipeline after the last update of bob.bio.baseThe latest version of bob.bio.base does not support to run any `simple` or `score-norm` pipeline. Below is the example error message:
```
▶ ./bin/bob bio pipeline simple rfw-idiap arcface-insightface -vvv -o idiap_arcface
Warning: entr...The latest version of bob.bio.base does not support to run any `simple` or `score-norm` pipeline. Below is the example error message:
```
▶ ./bin/bob bio pipeline simple rfw-idiap arcface-insightface -vvv -o idiap_arcface
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/local/scratch/linghu/miniconda3/envs/bob.paper.frice2/lib/python3.8/site-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/local/scratch/linghu/miniconda3/envs/bob.paper.frice2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2450, in load
return self.resolve()
File "/local/scratch/linghu/miniconda3/envs/bob.paper.frice2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2456, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/__init__.py", line 1, in <module>
from . import ( # noqa: F401
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/algorithm/__init__.py", line 1, in <module>
from .Algorithm import Algorithm
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/algorithm/Algorithm.py", line 10, in <module>
from .. import utils
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/utils/__init__.py", line 12, in <module>
from .io import * # noqa: F401,F403
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/utils/io.py", line 13, in <module>
from bob.bio.base import database
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/database/__init__.py", line 1, in <module>
from . import filelist # noqa: F401
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/database/filelist/__init__.py", line 2, in <module>
from .query import FileListBioDatabase
File "/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/database/filelist/query.py", line 10, in <module>
from .. import BioFile, ZTBioDatabase
ImportError: cannot import name 'BioFile' from partially initialized module 'bob.bio.base.database' (most likely due to a circular import) (/local/scratch/linghu/bob.bio.face/src/bob.bio.base/bob/bio/base/database/__init__.py)
```https://gitlab.idiap.ch/bob/bob.pad.face/-/issues/46Nightlies failing because of this one2022-05-02T18:15:24ZTiago de Freitas PereiraNightlies failing because of this one
https://gitlab.idiap.ch/bob/nightlies/-/jobs/266637
```
======================================================================
FAIL: Failure: AssertionError (File: not found: /Users/gitlab/builds/6eE8C-FW/1/bob/nightlies/miniconda/co...
https://gitlab.idiap.ch/bob/nightlies/-/jobs/266637
```
======================================================================
FAIL: Failure: AssertionError (File: not found: /Users/gitlab/builds/6eE8C-FW/1/bob/nightlies/miniconda/conda-bld/bob.pad.face_1651233096650/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.9/site-packages/bob/pad/face/test/data/real_client001_android_SD_scene01.mp4)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/gitlab/builds/6eE8C-FW/1/bob/nightlies/miniconda/conda-bld/bob.pad.face_1651233096650/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.9/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
```
I think it's just an issue with the MANIFEST.https://gitlab.idiap.ch/bob/nightlies/-/issues/62Nightlies on stages2023-05-11T09:17:24ZTiago de Freitas PereiraNightlies on stagesHi,
I think would be better to split this nightly builds in stages where in each stage we have a set of key packages.
For instance, in the first stage we could have the core packages.
- bob/bob.extension
- bob/bob.io.base
- bob/bob....Hi,
I think would be better to split this nightly builds in stages where in each stage we have a set of key packages.
For instance, in the first stage we could have the core packages.
- bob/bob.extension
- bob/bob.io.base
- bob/bob.measure
- bob/bob.learn.em
- bob/bob.pipelines
So, if one of these five packages (only five :-) ) fails we need to act immediately.
In the second, third or N-th stage we could pack a set of packages to be nightly executed. For instance, we could have a nightly for biometrics with the packages:
- bob/bob.bio.base
- bob/bob.bio.face
- bob/bob.bio.video
- bob/bob.bio.vein
- bob/bob.bio.spear
- bob/bob.pad.base
- bob/bob.pad.face
If one of these packages fails, we don't need to act urgently.
what do you think?
PS: this was @andre.anjos ideahttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/45ISV dask implementation is not efficient and workers run out of memory2022-05-09T13:00:47ZAmir MOHAMMADIISV dask implementation is not efficient and workers run out of memoryThe Great DeprecationAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.bio.spear/-/issues/40Re-implement i-vector and plda2022-04-27T20:16:19ZAmir MOHAMMADIRe-implement i-vector and pldadepends on https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/44depends on https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/44The Great DeprecationYannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/44Re-implement i-vector and plda2022-04-27T20:16:20ZAmir MOHAMMADIRe-implement i-vector and pldaThe Great DeprecationTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pad.face/-/issues/45Quality measure Python implementation2022-06-03T15:09:41ZAmir MOHAMMADIQuality measure Python implementationAlso code needs to be re-written to run fast.Also code needs to be re-written to run fast.The Great Deprecationhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/181re-enable cllr tests2022-05-12T13:36:23ZAmir MOHAMMADIre-enable cllr testsafter https://gitlab.idiap.ch/bob/bob.measure/-/issues/67 is resolved
by reverting https://gitlab.idiap.ch/bob/bob.bio.base/-/commit/7bee57cbfd1e98b2bc4f6e0044eeb178fdf766b4after https://gitlab.idiap.ch/bob/bob.measure/-/issues/67 is resolved
by reverting https://gitlab.idiap.ch/bob/bob.bio.base/-/commit/7bee57cbfd1e98b2bc4f6e0044eeb178fdf766b4The Great Deprecationhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/67rocch and calibration implementations are missing2022-05-12T13:36:56ZAmir MOHAMMADIrocch and calibration implementations are missingsince we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106since we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106The Great Deprecationhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/80Cannot specify original directory and extension for most of the databases any...2022-05-25T09:16:16ZManuel Günthersiebenkopf@googlemail.comCannot specify original directory and extension for most of the databases anymoreWhile in previous database implementations, it was relatively straightforward to utilize the database interface in order to load pre-extracted features, in the new database interfaces this is no longer possible, for two reasons:
1. Ther...While in previous database implementations, it was relatively straightforward to utilize the database interface in order to load pre-extracted features, in the new database interfaces this is no longer possible, for two reasons:
1. There is no simple way of providing the database interface with the directory where to read the data from. For example, in an old interface (LFW), we can still set:https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/c7ee7213f83f62b1e36685290e1defd10fea2c20/bob/bio/face/database/lfw.py#L90, while this option does no longer exist in newer interfaces: https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/c7ee7213f83f62b1e36685290e1defd10fea2c20/bob/bio/face/database/scface.py#L30, although it should be relatively straightforward to implement that since the default value is used just a few lines below: https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/c7ee7213f83f62b1e36685290e1defd10fea2c20/bob/bio/face/database/scface.py#L47. It should be simple to re-expose this option to the user.
2. The filename extension is by default empty: https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/c7ee7213f83f62b1e36685290e1defd10fea2c20/bob/bio/face/database/scface.py#L50, and there is no possibility to change that on the constructor. But even if we would expose the extension similarly to the directory name, we would still be in trouble since the default loader just **appends** the extension rather than ** replacing** it: https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/406f2da1faadacd4d4fe4e36e5a0010d78557513/bob/bio/base/database/csv_dataset.py#L145
The main issue with 2. is that someone has decided to ignore our old behavior to store keys without filename extension, and just added the original extension to the key. For example, running:
```
import bob.bio.face
db = bob.bio.face.database.SCFaceDatabase(protocol="far")
samples = db.all_samples()
print(samples[0].key)
```
will print `filename.JPG` instead of `filename` (without extension).
So, what I would propose is (the least amount of changes, anything else would require to re-create all the CSV-based databases):
- [ ] Expose the `original_directory` and `original_extension` parameters for all databases to the user, keeping their default values as they currently are.
- [ ] Change the line in https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/406f2da1faadacd4d4fe4e36e5a0010d78557513/bob/bio/base/database/csv_dataset.py#L145 to replace the extension with the given one (if one is given) rather than appending it to the filename.
Any objection?