bob issueshttps://gitlab.idiap.ch/groups/bob/-/issues2022-03-24T15:13:17Zhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/39gmm methods and e_step/m_step functions do not share code but implement the s...2022-03-24T15:13:17ZAmir MOHAMMADIgmm methods and e_step/m_step functions do not share code but implement the same thingAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/38More dask tests2022-03-24T18:21:54ZAmir MOHAMMADIMore dask tests- [ ] There are no dask array as input tests for gmms
- [ ] Dask tests should run under multiprocessing of distributed package to make sure real-world conditions are simulated.- [ ] There are no dask array as input tests for gmms
- [ ] Dask tests should run under multiprocessing of distributed package to make sure real-world conditions are simulated.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/37Excess memory usage in kmeans training2022-03-22T17:34:30ZAmir MOHAMMADIExcess memory usage in kmeans trainingWhen training on voxforge and 256 GMMs with dask partition size of 200, I get this error:
```
bob.learn.em/bob/learn/em/k_means.py", line 78, in e_step
np.eye(n_clusters)[closest_k_indices][:, :, None] * data[:, None],
numpy.core._ex...When training on voxforge and 256 GMMs with dask partition size of 200, I get this error:
```
bob.learn.em/bob/learn/em/k_means.py", line 78, in e_step
np.eye(n_clusters)[closest_k_indices][:, :, None] * data[:, None],
numpy.core._exceptions.MemoryError: Unable to allocate 7.39 GiB for an array with shape (64546, 256, 60) and data type float64
```Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/77Output of `dataset.all_samples` is inconsistent2022-03-10T18:55:22ZManuel Günthersiebenkopf@googlemail.comOutput of `dataset.all_samples` is inconsistentThe output of the method `all_samples` of different databases returns different things. While the default `CSVDataset` https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/997e6d6dda44c928c1792518a2b625726efde0e1/bob/bio/base/database/csv_dat...The output of the method `all_samples` of different databases returns different things. While the default `CSVDataset` https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/997e6d6dda44c928c1792518a2b625726efde0e1/bob/bio/base/database/csv_dataset.py#L744 returns a list of `Sample` (more precisely a list of `DelayedSample`), some other datasets implemented in here return a list of `SampleSet`. Examples are:
* https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/38a910ac1df0ba14e8262f957ae0e666a3e2f616/bob/bio/face/database/ijbc.py#L296
* https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/38a910ac1df0ba14e8262f957ae0e666a3e2f616/bob/bio/face/database/rfw.py#L424
* https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/38a910ac1df0ba14e8262f957ae0e666a3e2f616/bob/bio/face/database/gbu.py#L238
But I am sure that I was missing some.
Is there any plan in changing this inconsistency? The name of the function suggests to extract a list of `Sample`, so we would likely want to adapt the implementations of the datasets listed here...https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/36get_centroids_distance gets called twice during e_step in kmeans2022-03-21T10:48:27ZAmir MOHAMMADIget_centroids_distance gets called twice during e_step in kmeanshttps://gitlab.idiap.ch/bob/bob.pad.base/-/issues/40Refactor vanilla pad script as reusable functions2022-03-03T17:24:16ZAmir MOHAMMADIRefactor vanilla pad script as reusable functionsAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.extension/-/issues/83Matplotlib base documentation directory has changed2022-03-01T08:23:13ZAndré AnjosMatplotlib base documentation directory has changedWe need to modify the base URL for the matplotlib manual at https://gitlab.idiap.ch/bob/bob.extension/-/blob/master/bob/extension/utils.py#L527 from `https://matplotlib.org/` to `https://matplotlib.org/stable/`. Builds are breaking on t...We need to modify the base URL for the matplotlib manual at https://gitlab.idiap.ch/bob/bob.extension/-/blob/master/bob/extension/utils.py#L527 from `https://matplotlib.org/` to `https://matplotlib.org/stable/`. Builds are breaking on this account:
https://gitlab.idiap.ch/bob/bob.ip.binseg/-/jobs/258731https://gitlab.idiap.ch/bob/bob.ip.facedetect/-/issues/13Nightlies failing because of this one2022-02-28T14:34:46ZTiago de Freitas PereiraNightlies failing because of this oneNow it is something related with TF
https://gitlab.idiap.ch/bob/nightlies/-/jobs/258558Now it is something related with TF
https://gitlab.idiap.ch/bob/nightlies/-/jobs/258558https://gitlab.idiap.ch/bob/bob.devtools/-/issues/93Missing CUDA overrides2022-02-28T14:34:46ZAmir MOHAMMADIMissing CUDA overridesJob [#258340](https://gitlab.idiap.ch/bob/bob.devtools/-/jobs/258340) failed for 93acc950ec94de1ff7d4629d8bdfbe9cf8c25c77:Job [#258340](https://gitlab.idiap.ch/bob/bob.devtools/-/jobs/258340) failed for 93acc950ec94de1ff7d4629d8bdfbe9cf8c25c77:Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/35Small changes to the new API needed2022-02-24T17:21:38ZAmir MOHAMMADISmall changes to the new API needed- [ ] rename kmeans_trainer parameter in GMMMachine to `kmeans_machine`
- [ ] make the Kmeans and GMM machines available in `bob.learn.em`.
- ~~[ ] The `__call__` methods are missing.~~
- [ ] load methods are missing
- [ ] setup.py requi...- [ ] rename kmeans_trainer parameter in GMMMachine to `kmeans_machine`
- [ ] make the Kmeans and GMM machines available in `bob.learn.em`.
- ~~[ ] The `__call__` methods are missing.~~
- [ ] load methods are missing
- [ ] setup.py requirements.txt and meta.yaml is not updated.
Some other things that I will add to this issue.
The changes don't need to be addressed now.https://gitlab.idiap.ch/bob/bob.fusion.base/-/issues/9Needs to be ported with the new bob.learn.em API2022-03-02T09:27:39ZAmir MOHAMMADINeeds to be ported with the new bob.learn.em APIJob [#258101](https://gitlab.idiap.ch/bob/bob.fusion.base/-/jobs/258101) failed for 506b8b66a584ffb37d6038b6a87a5e40510f2142:Job [#258101](https://gitlab.idiap.ch/bob/bob.fusion.base/-/jobs/258101) failed for 506b8b66a584ffb37d6038b6a87a5e40510f2142:Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.bio.vein/-/issues/26Fails on MacOS Arm with AttributeError: module 'numpy' has no attribute 'floa...2022-03-03T14:22:23ZAmir MOHAMMADIFails on MacOS Arm with AttributeError: module 'numpy' has no attribute 'float128'Job [#256859](https://gitlab.idiap.ch/bob/bob.bio.vein/-/jobs/256859) failed for 34ddbc6fb35136fcd3cdf7ad6b87296c023b2c64:
```
======================================================================
ERROR: bob.bio.vein.tests.test.test_mas...Job [#256859](https://gitlab.idiap.ch/bob/bob.bio.vein/-/jobs/256859) failed for 34ddbc6fb35136fcd3cdf7ad6b87296c023b2c64:
```
======================================================================
ERROR: bob.bio.vein.tests.test.test_mask_to_image
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/gitlab/builds/sAZPGTqW/0/bob/bob.bio.vein/miniconda/conda-bld/bob.bio.vein_1644593419589//lib/python3.9/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/gitlab/builds/sAZPGTqW/0/bob/bob.bio.vein/miniconda/conda-bld/bob.bio.vein_1644593419589//lib/python3.9/site-packages/bob/bio/vein/tests/test.py", line 474, in test_mask_to_image
_check_float(32)
File "/Users/gitlab/builds/sAZPGTqW/0/bob/bob.bio.vein/miniconda/conda-bld/bob.bio.vein_1644593419589//lib/python3.9/site-packages/bob/bio/vein/tests/test.py", line 470, in _check_float
conv = preprocessor_utils.mask_to_image(sample, 'float%d' % n)
File "/Users/gitlab/builds/sAZPGTqW/0/bob/bob.bio.vein/miniconda/conda-bld/bob.bio.vein_1644593419589//lib/python3.9/site-packages/bob/bio/vein/preprocessor/utils.py", line 158, in mask_to_image
elif dtype in (numpy.float32, numpy.float64, numpy.float128):
File "/Users/gitlab/builds/sAZPGTqW/0/bob/bob.bio.vein/miniconda/conda-bld/bob.bio.vein_1644593419589//lib/python3.9/site-packages/numpy/__init__.py", line 313, in __getattr__
raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float128'
```https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/34Does not work with h5py 32022-02-21T18:23:19ZAmir MOHAMMADIDoes not work with h5py 3Job [#257793](https://gitlab.idiap.ch/bob/bob.learn.em/-/jobs/257793) failed for 462d8bda27b8bef01a11f187fbb7ade153a650a0:Job [#257793](https://gitlab.idiap.ch/bob/bob.learn.em/-/jobs/257793) failed for 462d8bda27b8bef01a11f187fbb7ade153a650a0:Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/33Nightlies failing because of this one2022-02-22T13:48:19ZTiago de Freitas PereiraNightlies failing because of this oneNightlies is failing because of this one.
The issue seems related to HDF5
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257644
https://gitlab.idiap.ch/bob/bob.learn.em/-/jobs/257680
Could you please look at that @ydayer ?
ThanksNightlies is failing because of this one.
The issue seems related to HDF5
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257644
https://gitlab.idiap.ch/bob/bob.learn.em/-/jobs/257680
Could you please look at that @ydayer ?
ThanksYannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/175Script for feature extraction from database2022-03-19T11:10:26ZManuel Günthersiebenkopf@googlemail.comScript for feature extraction from databaseIn many cases, we would just want to have a script to extract the features for all samples of our database (using a specifiable ``Transformer``), so that we can use them in a different process. Currently, there is no such script availabl...In many cases, we would just want to have a script to extract the features for all samples of our database (using a specifiable ``Transformer``), so that we can use them in a different process. Currently, there is no such script available.
I would propose to add a script as follows:
```
import argparse
import os
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
description='Extract features from the given dataset'
)
parser.add_argument("--transformer", "-e", required=True, help="Select the transformer to be used")
parser.add_argument("--dataset", "-d", required=True, help="Select the dataset from which to extract features")
parser.add_argument("--output-directory", "-o", required=True, help="Select the directory where to write the data to")
args = parser.parse_args()
import bob.bio.base
import bob.core
import bob.io.base
logger = bob.core.log.setup("bob.paper.osijbc")
bob.core.log.set_verbosity_level(logger, 2)
database = bob.bio.base.load_resource(args.dataset, "database")
transformer = bob.bio.base.load_resource(args.transformer, "transformer")
for idx, samples in enumerate(database.all_samples()):
logger.info('Extracting features for sample', )
features = transformer.transform(samples)
for feature in features:
output = os.path.join(args.output_directory, feature.key + ".hdf5")
logger.debug('Writing file', output)
bob.io.base.save(feature.data, output, True)
```
To be consistent with our other scripts, I would recommend to use `click` instead of `argparse`. Unfortunately, I am not familiar with click and I have no time to learn how to implement click commands right now. Would anyone else -- with more experience with `click` take this over?Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.io.base/-/issues/22Does not compile with hdf5 1.122022-02-21T11:37:29ZTiago de Freitas PereiraDoes not compile with hdf5 1.12Compilation issue with HDF5
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257523/raw
Probably related to these bumps
https://gitlab.idiap.ch/bob/bob.devtools/-/merge_requests/273
```
/scratch/builds/bob/nightlies/src/bob/bob.io.base/...Compilation issue with HDF5
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257523/raw
Probably related to these bumps
https://gitlab.idiap.ch/bob/bob.devtools/-/merge_requests/273
```
/scratch/builds/bob/nightlies/src/bob/bob.io.base/bob/io/base/cpp/HDF5Group.cpp: In member function 'herr_t bob::io::base::detail::hdf5::Group::iterate_callback(hid_t, const char*, const H5L_info2_t*)':
/scratch/builds/bob/nightlies/src/bob/bob.io.base/bob/io/base/cpp/HDF5Group.cpp:88:73: error: too few arguments to function 'herr_t H5Oget_info_by_name3(hid_t, const char*, H5O_info2_t*, unsigned int, hid_t)'
88 | herr_t status = H5Oget_info_by_name(self, name, &obj_info, H5P_DEFAULT);
| ^
In file included from /scratch/builds/bob/nightlies/miniconda/conda-bld/bob.io.base_1645283626826/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/H5Apublic.h:22,
from /scratch/builds/bob/nightlies/miniconda/conda-bld/bob.io.base_1645283626826/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/hdf5.h:23,
from /scratch/builds/bob/nightlies/src/bob/bob.io.base/bob/io/base/include/bob.io.base/HDF5Group.h:15,
from /scratch/builds/bob/nightlies/src/bob/bob.io.base/bob/io/base/cpp/HDF5Group.cpp:18:
/scratch/builds/bob/nightlies/miniconda/conda-bld/bob.io.base_1645283626826/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/H5Opublic.h:557:15: note: declared here
557 | H5_DLL herr_t H5Oget_info_by_name3(hid_t loc_id, const char *name, H5O_info2_t *oinfo, unsigned fields,
| ^~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/bob_io_base.dir/build.make:160: CMakeFiles/bob_io_base.dir/scratch/builds/bob/nightlies/src/bob/bob.io.base/bob/io/base/cpp/HDF5Group.cpp.o] Error 1
```Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pad.face/-/issues/43Nightlies failing2022-02-22T17:55:07ZTiago de Freitas PereiraNightlies failingNightlies failing because of this one
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257429
Can someone have a look on that?
Thanks
ping @amohammadi @ageorgeNightlies failing because of this one
https://gitlab.idiap.ch/bob/nightlies/-/jobs/257429
Can someone have a look on that?
Thanks
ping @amohammadi @ageorgeAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/174Nightlies failing here2022-02-16T09:30:59ZTiago de Freitas PereiraNightlies failing herePLDA was temporarily removed from `bob.learn.em` with https://gitlab.idiap.ch/bob/bob.learn.em/-/merge_requests/42
we need to remove it from here too.PLDA was temporarily removed from `bob.learn.em` with https://gitlab.idiap.ch/bob/bob.learn.em/-/merge_requests/42
we need to remove it from here too.https://gitlab.idiap.ch/bob/bob.ip.facedetect/-/issues/12Nightlies failling here2022-02-15T15:40:38ZTiago de Freitas PereiraNightlies failling hereSomething with boosting.
I don't understand why this is failing now.
Needs investigation
https://gitlab.idiap.ch/bob/nightlies/-/pipelines/58308Something with boosting.
I don't understand why this is failing now.
Needs investigation
https://gitlab.idiap.ch/bob/nightlies/-/pipelines/58308https://gitlab.idiap.ch/bob/bob.extension/-/issues/82Nightlies broken - scipy intersphinx is broken2022-03-22T12:57:35ZTiago de Freitas PereiraNightlies broken - scipy intersphinx is brokenWe are getting 404 while fetching the scipy intersphinx.
There are some discussion going on here
https://github.com/scipy/scipy/issues/14267
and here
https://github.com/scipy/scipy/issues/15574
Let's see how it goesWe are getting 404 while fetching the scipy intersphinx.
There are some discussion going on here
https://github.com/scipy/scipy/issues/14267
and here
https://github.com/scipy/scipy/issues/15574
Let's see how it goes