bob issueshttps://gitlab.idiap.ch/groups/bob/-/issues2020-06-24T11:16:38Zhttps://gitlab.idiap.ch/bob/bob.devtools/-/issues/54undefined variable in build.py2020-06-24T11:16:38ZVincent POLLETundefined variable in build.pyHi, the variable `path` is undefined [here](https://gitlab.idiap.ch/bob/bob.devtools/-/blob/master/bob/devtools/build.py#L651)
It must have been refactored during the recent work on `base_build` but I could not find where exactly. In ht...Hi, the variable `path` is undefined [here](https://gitlab.idiap.ch/bob/bob.devtools/-/blob/master/bob/devtools/build.py#L651)
It must have been refactored during the recent work on `base_build` but I could not find where exactly. In https://gitlab.idiap.ch/bob/bob.devtools/-/commit/7ead642b2393bfc750d80aecc2873956575aabb7 it is changed to `recipe_dir`
It causes the pipeline of https://gitlab.idiap.ch/bob/bob.conda/-/merge_requests/444 to fail.
Thankshttps://gitlab.idiap.ch/bob/bob.extension/-/issues/75rc subsystem is not very testable2020-07-22T08:50:49ZAndré Anjosrc subsystem is not very testableI'm trying to do tests on libraries that work with the RC subsystem.
One of the problems I'm facing is that it is difficult to mock it - once `bob.extension` is imported, the contents of the RC file is loaded and there seems to be no ea...I'm trying to do tests on libraries that work with the RC subsystem.
One of the problems I'm facing is that it is difficult to mock it - once `bob.extension` is imported, the contents of the RC file is loaded and there seems to be no easy way to affect such a value in a cross-module way.
Ideally, what you'd like to do is something like this:
```python
@mock_bobs_rc('var1', 'override-value1')
def test():
# now, code that uses bob.extension.rc will use the mock value
# notice that the code is not necessarily here, but may be used several layers down
```
Can you think of an easy solution to this?André AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/134`check_existence` flag incorrectly handled in filelistdatabase query2020-06-03T08:58:06ZManuel Günthersiebenkopf@googlemail.com`check_existence` flag incorrectly handled in filelistdatabase queryIn https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/3efccd3b637ee73ec68ed0ac5fde2667a943bd6e/bob/bio/base/database/filelist/query.py#L833, the `check_existence` flag is said to be ignored when multiple original extensions are specified, w...In https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/3efccd3b637ee73ec68ed0ac5fde2667a943bd6e/bob/bio/base/database/filelist/query.py#L833, the `check_existence` flag is said to be ignored when multiple original extensions are specified, while it is actually not ignored.
Also, when only a single extension is specified, the `check_existence` flag is tested incorrectly.Manuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.comhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/18SampleBatch design issues2020-07-22T14:26:59ZTiago de Freitas PereiraSampleBatch design issuesHi,
Although `SampleBatch` brings convenience and efficiency, it forces us to develop transformers that are compatible with it.
Imagine the simple transformer bellow:
```python
class FakeTransformer(TransformerMixin, BaseEstimator):
...Hi,
Although `SampleBatch` brings convenience and efficiency, it forces us to develop transformers that are compatible with it.
Imagine the simple transformer bellow:
```python
class FakeTransformer(TransformerMixin, BaseEstimator):
def fit(self, X, y=None):
return self
def transform(self, X):
return X + 1
def _more_tags(self):
return {"stateless": True, "requires_fit": False}
```
I can easily use it with numpy arrays as input.
```python
transformer = FakeTransformer()
X = np.zeros(shape=(3, 160, 160))
transformed_X = transformer.transform(X)
```
However, I run into problems once I wrap it as a sample
```python
sample = Sample(X)
transformer_sample = wrap(["sample"], transformer)
my_beautiful_sample = [s.data for s in transformer_sample.transform([sample])]
# THIS DOESN'T WORK
```
With this wrap, the input `X` of `FakeTransformer.transform` will be `SampleBatch` and not numpy array.
Hence, I can't do `X+1`.
I can approach this issue in my transformer by doing this:
```python
def transform(self, X):
X = np.asarray(X)
return X + 1
```
However, this is a blocker if we want to use estimators developed by other people outside of our circle.
Do you think it is sensible to have `X` wrapped as a `SampleBatch` once SampleTransform is used?
It breaks encapsulation.
Thankshttps://gitlab.idiap.ch/bob/bob.db.morph/-/issues/3Database.zprobes and Database.treferences fraction takes a fraction of....2020-12-22T08:28:23ZTiago de Freitas PereiraDatabase.zprobes and Database.treferences fraction takes a fraction of.......absulute values.
This function should take fraction from each cohort....absulute values.
This function should take fraction from each cohort.https://gitlab.idiap.ch/bob/bob.db.morph/-/issues/2This dataset has wrong annotations2020-12-22T08:28:41ZTiago de Freitas PereiraThis dataset has wrong annotationsHey @ydayer,
I'm rearranging world, dev, and eval in this dataset.
Have you noticed that the metadata is inconsistent?
For instance, if you do:
```python
>>> dataframe[dataframe.id_num==286810]
id_num picture_num dob ...Hey @ydayer,
I'm rearranging world, dev, and eval in this dataset.
Have you noticed that the metadata is inconsistent?
For instance, if you do:
```python
>>> dataframe[dataframe.id_num==286810]
id_num picture_num dob doa race gender age photo
37850 286810 1 04/04/1986 05/11/2006 B M 20 Album2/286810_01M20.JPG
37851 286810 2 04/04/1986 08/16/2006 A M 20 Album2/286810_02M20.JPG
37849 286810 0 04/04/1986 01/24/2006 H M 19 Album2/286810_00M19.JPG
```
```python
>>> dataframe[dataframe.id_num==295087]
id_num picture_num dob doa race gender age photo
39551 295087 0 05/18/1960 10/23/2006 A M 46 Album2/295087_00M46.JPG
39552 295087 1 05/18/1960 10/25/2006 H M 46 Album2/295087_01M46.JPG
````
```python
dataframe[dataframe.id_num==328749]
id_num picture_num dob doa race gender age photo
50810 328749 0 07/28/1971 05/12/2006 W M 34 Album2/328749_00M34.JPG
50811 328749 1 07/28/1971 05/19/2007 A M 35 Album2/328749_01M35.JPG
```
There are several more exampleshttps://gitlab.idiap.ch/bob/bob.ip.gabor/-/issues/5JetStatistics divides by the wrong value2020-06-03T06:45:13ZManuel Günthersiebenkopf@googlemail.comJetStatistics divides by the wrong valueIn line https://gitlab.idiap.ch/bob/bob.ip.gabor/-/blob/94bd69147ca4450ab9f975efab5ee31bbf3edefd/bob/ip/gabor/cpp/JetStatistics.cpp#L145 we divide by `m_varAbs(j)`, but we'd need to divide by `m_meanAbs(j)`.In line https://gitlab.idiap.ch/bob/bob.ip.gabor/-/blob/94bd69147ca4450ab9f975efab5ee31bbf3edefd/bob/ip/gabor/cpp/JetStatistics.cpp#L145 we divide by `m_varAbs(j)`, but we'd need to divide by `m_meanAbs(j)`.Manuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.comhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/36outdated `baselines.py` script remained in setup.py2020-10-08T07:25:59ZManuel Günthersiebenkopf@googlemail.comoutdated `baselines.py` script remained in setup.pySee here: https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/117ed4a764bac88c9e0d3c5b27c8c36d5cf87a61/setup.py#L106
When running `baselines.py`, I get the following error:
```
$ baselines.py -h
...
ModuleNotFoundError: No module named 'bob...See here: https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/117ed4a764bac88c9e0d3c5b27c8c36d5cf87a61/setup.py#L106
When running `baselines.py`, I get the following error:
```
$ baselines.py -h
...
ModuleNotFoundError: No module named 'bob.bio.face.script.baselines'
```Bob 9.0.0https://gitlab.idiap.ch/bob/bob.devtools/-/issues/53Implement hot-fix to repo indexing2020-07-29T14:07:12ZAndré AnjosImplement hot-fix to repo indexingTo fix several issues that we are having with our conda channel, I am going to
implement [this hotfixing mechanism](https://github.com/AnacondaRecipes/repodata-hotfixes)
done for the defaults channel for our channel as well.
This will:
...To fix several issues that we are having with our conda channel, I am going to
implement [this hotfixing mechanism](https://github.com/AnacondaRecipes/repodata-hotfixes)
done for the defaults channel for our channel as well.
This will:
1. remove the need of moving of broken packages to our archive channel. Instead, we will keep the package in the same place but remove it from the index.
2. allow us to fix broken packages in our channel. Like fixing bob.bio.base to make
sure it does not get installed with `numpy>=1.8`.
3. allow us to temporarily add packages (like mac versions of pytorch and torchvision)
in our channel to fix our problems and remove them from channel index once the
defaults channel catches up.
But to implement this care is needed from @bob users when they try to export an environment.
In summary, you should avoid mixing `conda env export` and `conda list --export --explicit`.
These 2 commands are designed in conda with two different goals and you should not use them
for other purposes. I will explain what to do below:
# Reproducibility and Repeatability of publications (bob.paper packages)
You should use `conda list --export --explicit` or even `conda list --export --explicit --md5`
to export your environment for other users to replicate your environment.
```
Save packages for future use:
conda list --export --explicit > package-list.txt
or
conda list --export --explicit --md5 > package-list.txt
Reinstall packages from an export file:
conda create -n myenv --file package-list.txt
```
This method is, of course, not bullet proof but it should work reliably.
If you use `conda env export`, the environment **will** most likely break in the future.
# Share current ongoing work/projects (bob.project packages)
Sometimes, you want to share a common conda environment between colleagues while a
project is continuing. You may even update this environment regularly.
For this purpose, you should use `conda env export` or even better,
create your `environment.yml` by hand. You may create environment files that work
both on Linux and mac.
```
Create the environment file either by hand or by
conda env export --file=environment.yml
Recreate the environment using:
conda env create --file=environment.yml
```
Expect this environment to become broken from time to time and it might need updates.
To avoid *some* breakage, do not pin the build strings, i.e.
instead of `bob.bio.base=4.1.0=py37h03d05df_0`, write `bob.bio.base=4.1.0`.
Also, you may want to only list your direct dependencies only.
Of course, you can choose to export to both formats in any scenario.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/17Memory error during serialization of large objects2020-05-25T08:48:53ZTiago de Freitas PereiraMemory error during serialization of large objectsThis is an issue that I'm facing for a while.
Now we are running our pipelines in large scale experiments (several thousands of images), the list of SampleSets that we are generating during `pipeline.transform` are getting BIG (>1GB) an...This is an issue that I'm facing for a while.
Now we are running our pipelines in large scale experiments (several thousands of images), the list of SampleSets that we are generating during `pipeline.transform` are getting BIG (>1GB) and this is raising some MemoryError Exceptions during serialization (even when we have enough memory).
This is very annoying, basically, I can't work with large datasets.
I managed to generate a very simple example describing this issue here: https://github.com/dask/distributed/issues/3806
I know we can change the serializer` dask-distributed` uses (https://distributed.dask.org/en/latest/serialization.html#use), but I'm not sure if this is the real problem.
However, I would like to propose a workaround that will slow down a bit the execution of experiments, but, at least, the code will not crash.
I would like to change the serialization behavior of DelayedSamples to this.
```python
class DelayedSample(_ReprMixin):
def __init__(self, load, parent=None, **kwargs):
self.load = load
if parent is not None:
_copy_attributes(self, parent.__dict__)
_copy_attributes(self, kwargs)
self._data = None
@property
def data(self):
"""Loads the data from the disk file."""
if self._data is None:
self._data = self.load()
return self._data
def __getstate__(self):
self._data = None
d = dict(self.__dict__)
return d
```
What do you think? ping @andre.anjos @amohammadi
ping @ydayer
thankshttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/16Run dask+SGE backend without SETSHELL = starvation2020-05-16T12:47:09ZTiago de Freitas PereiraRun dask+SGE backend without SETSHELL = starvationIt's being a while we have this issue, but I've never materialized it here.
When a script that uses dask+SGE backend is executed at Idiap without a `SETSHELL grid`, the script hangs forever waiting for a worker.
We need to, i-) either t...It's being a while we have this issue, but I've never materialized it here.
When a script that uses dask+SGE backend is executed at Idiap without a `SETSHELL grid`, the script hangs forever waiting for a worker.
We need to, i-) either throw an exception once `qsub` fails to be executed or ii-) do like it's done in gridtk (triggers this automatically).https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/15Sample-based pipelines inefficiencies2020-06-02T08:04:29ZAmir MOHAMMADISample-based pipelines inefficienciesThis is a generic issue that I am raising that I believe we will face moving forward.
The biggest issue that I have found with our sample-based approach is when you have to concatenate samples to make a big array for processing steps suc...This is a generic issue that I am raising that I believe we will face moving forward.
The biggest issue that I have found with our sample-based approach is when you have to concatenate samples to make a big array for processing steps such as `.fit` methods.
The reason for this is that we are looking at samples individually, **even though they might have come from a bigger array**.
Let me demonstrate this with an example:
[sample_stacking_issue.html](/uploads/689b77611dd7d51685b43862be9c2686/sample_stacking_issue.html)
or [sample_stacking_issue.ipynb](/uploads/d2139734a2dae8833104c46b0022b2eb/sample_stacking_issue.ipynb)https://gitlab.idiap.ch/bob/bob.io.video/-/issues/16bob.io.video.reader class is not pickable2020-05-04T18:23:20ZAmir MOHAMMADIbob.io.video.reader class is not pickableSee: https://github.com/cloudpipe/cloudpickle/issues/363See: https://github.com/cloudpipe/cloudpickle/issues/363Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/14Stacking raw data2020-05-05T15:29:09ZTiago de Freitas PereiraStacking raw dataHi,
I haven't noticed that before merging, but the way this `DelayedSamplesCall` handles data is not super convenient in terms of memory usage, don't you think? https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/bob/pipelines/wrapp...Hi,
I haven't noticed that before merging, but the way this `DelayedSamplesCall` handles data is not super convenient in terms of memory usage, don't you think? https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/bob/pipelines/wrappers.py#L57
Data are stacked at the very beginning of the pipeline, which can be huge in size (e.g raw images/video).
The way we had it before https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/26, was more convenient.
This stacking was done only when necessary (once data is "more preprocessed").
Is there any reason to be like that?
Thankshttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/35Nightlies failing because of this one2020-04-29T06:52:54ZTiago de Freitas PereiraNightlies failing because of this oneThis MR broke https://gitlab.idiap.ch/bob/bob.ip.gabor/-/merge_requests/12
What needs to be done is to cherry-pick these commits 8cf2dd5544957a7c86a3245ba8e5ca65f9c7a9ca and e9e7bbb7e80ea01e37cb54ecc04ce2020e09b25d, and merge them to ma...This MR broke https://gitlab.idiap.ch/bob/bob.ip.gabor/-/merge_requests/12
What needs to be done is to cherry-pick these commits 8cf2dd5544957a7c86a3245ba8e5ca65f9c7a9ca and e9e7bbb7e80ea01e37cb54ecc04ce2020e09b25d, and merge them to masterTiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.learn.mlp/-/issues/11DEPRECATION2021-09-17T14:19:31ZTiago de Freitas PereiraDEPRECATIONThis package will be deprecated in favor of Tensorflow/PyTorchThis package will be deprecated in favor of Tensorflow/PyTorchBob 9.0.0Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.ip.flandmark/-/issues/9DEPRECATION2020-11-04T11:23:18ZTiago de Freitas PereiraDEPRECATIONThis package will be deprecatedThis package will be deprecatedBob 9.0.0https://gitlab.idiap.ch/bob/bob.ip.facedetect/-/issues/8DEPRECATION2020-11-05T13:42:10ZTiago de Freitas PereiraDEPRECATIONThis package will be deprecatedThis package will be deprecatedBob 9.0.0https://gitlab.idiap.ch/bob/bob.learn.boosting/-/issues/3DEPRECATION2020-11-05T13:42:11ZTiago de Freitas PereiraDEPRECATIONThis package will be deprecatedThis package will be deprecatedBob 9.0.0https://gitlab.idiap.ch/bob/bob.learn.libsvm/-/issues/12DEPRECATION2021-09-17T14:19:31ZTiago de Freitas PereiraDEPRECATIONThis package will be deprecatedThis package will be deprecatedBob 9.0.0Amir MOHAMMADIAmir MOHAMMADI