bob issueshttps://gitlab.idiap.ch/groups/bob/-/issues2024-03-21T10:21:20Zhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/29jman (gridtk) like interface for submitting dask jobs2024-03-21T10:21:20ZAmir MOHAMMADIjman (gridtk) like interface for submitting dask jobsWe need:
1. A command that automatically creates a dask client for us to be used for SGE submission.
2. A history of the commands that were executed.
3. An automatic tracking of dask logs.We need:
1. A command that automatically creates a dask client for us to be used for SGE submission.
2. A history of the commands that were executed.
3. An automatic tracking of dask logs.Bob 9.0.0https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/21Who is Mario?2024-01-08T13:58:44ZTiago de Freitas PereiraWho is Mario?There are several places in the code where we alias `bob.pipelines` as Mario.
Well, it was a good joke at the beginning but now we need to think if we want to keep this.There are several places in the code where we alias `bob.pipelines` as Mario.
Well, it was a good joke at the beginning but now we need to think if we want to keep this.Tiago de Freitas PereiraTiago de Freitas Pereirahttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/43Remove Bob Extension as dependency2024-01-08T13:57:32ZAndré MAYORAZRemove Bob Extension as dependencyBob extension has to be removed. It is used at three places in this package :
- [x] bob.pipelines/doc/conf.py
- To load the association list for the packages for intersphinx. We have to see if it is better to do the association between ...Bob extension has to be removed. It is used at three places in this package :
- [x] bob.pipelines/doc/conf.py
- To load the association list for the packages for intersphinx. We have to see if it is better to do the association between the package and their URL directly or any other solution.
- [x] bob.pipelines/src/bob/pipelines/distributed/sge.py
- To load the rc configuration. This can be replaced by the [exposed](https://gitlab.idiap.ch/bob/exposed) package
- [x] bob.pipelines/src/bob/pipelines/datasets.py
- To list the files and folders inside a folder or a tarball and search for files either in a file structure or in a tarball.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.paper.makeup_aim/-/issues/1Please don't encourage bad practices of Bob's global configuration system.2023-12-07T08:03:25ZAmir MOHAMMADIPlease don't encourage bad practices of Bob's global configuration system.Please see:
https://www.idiap.ch/software/bob/docs/bob/docs/stable/bob/bob.extension/doc/rc.html
and https://groups.google.com/forum/#!topic/bob-devel/hA1sx5iAj9UPlease see:
https://www.idiap.ch/software/bob/docs/bob/docs/stable/bob/bob.extension/doc/rc.html
and https://groups.google.com/forum/#!topic/bob-devel/hA1sx5iAj9UKetan KotwalKetan Kotwalhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/195download_file extracts the archive file every time it is called.2023-10-31T13:42:42ZYannick DAYERdownload_file extracts the archive file every time it is called.When the `extract` flag is set in [`download_file`](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/v8.0.0/src/bob/bio/base/database/utils.py?ref_type=tags#L474), the archive file is extracted at every call, even if the first call that d...When the `extract` flag is set in [`download_file`](https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/v8.0.0/src/bob/bio/base/database/utils.py?ref_type=tags#L474), the archive file is extracted at every call, even if the first call that downloaded the archive already extracted it. The issue is that for some archives and on some systems, this extraction takes time (more than 10s).
Choices:
- Keep the extraction even if the archive was not re-downloaded:
- Takes time at the start of every run.
- Ensures the extracted files are correct and were not modified.
- Only extract the archive just after downloading it:
- If the extracted files are modified between runs, the next run will use those.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/193Add a `--score-column` option in metrics and figures2023-08-22T07:59:18ZYannick DAYERAdd a `--score-column` option in metrics and figuresIn order to process score files with multiple score columns, we should have a way to select a specific column that contains the scores.
``` sh
bob bio metrics /path/to/the/file --score-column "fusion_score"
```
(Until now, we needed to...In order to process score files with multiple score columns, we should have a way to select a specific column that contains the scores.
``` sh
bob bio metrics /path/to/the/file --score-column "fusion_score"
```
(Until now, we needed to change both columns' headers to do that).Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob/-/issues/274Switch to new CI/CD configuration2023-06-13T14:59:52ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [ ] Modify `pyproject.toml`,
- Add information from `setup.py`,
- [ ] Empty `setup.py`,
- Leave the call to `setup()` for compatibility,
- [ ] Mo...We need to adapt this package to the new CI/CD and package format using citools:
- [ ] Modify `pyproject.toml`,
- Add information from `setup.py`,
- [ ] Empty `setup.py`,
- Leave the call to `setup()` for compatibility,
- [ ] Modify `conda/meta.yaml`,
- [ ] Remove test and documentation commands,
- [ ] Import names from `pyproject.toml`,
- [ ] Modify `.gitlab-ci.yml`,
- Use the nicer format instead of the URL,
- [ ] Add the project path to PATH in `doc/conf.py`,
- [ ] Remove `version.txt` and put the version in `pyproject.toml`.
You can look at bob.learn.em!68 for an example of a ported package.
*Note: This one might require special care compared to other ported packages as it is not a simple package*Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/nightlies/-/issues/64Pipeline passes even when some sub-pipeline fails2023-05-11T09:47:04ZYannick DAYERPipeline passes even when some sub-pipeline failsWhen the bob.bio.face pipeline fails (e.g. [on May the 4th](https://gitlab.idiap.ch/bob/bob.bio.face/-/pipelines/73071)), the [nightlies pipeline](https://gitlab.idiap.ch/bob/nightlies/-/pipelines/73059) does not care that it failed (it ...When the bob.bio.face pipeline fails (e.g. [on May the 4th](https://gitlab.idiap.ch/bob/bob.bio.face/-/pipelines/73071)), the [nightlies pipeline](https://gitlab.idiap.ch/bob/nightlies/-/pipelines/73059) does not care that it failed (it even marks it as succeeded), and continues executing.
I'm expecting the nightlies' pipeline to stop (or at least show that one job failed).
This is not critical as we receive a notification anyway, at the level of the package (here bob.bio.face).https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/190SampleSet missing attributes 'key'2023-04-04T17:55:49ZChristophe ECABERTSampleSet missing attributes 'key'With the updated version of the `CSVDatabase` the `references` and `probes` are now `SampleSet` objects. Special care is taken to make sure the required attributes are defined for samples within them.
However there is no check for `Sam...With the updated version of the `CSVDatabase` the `references` and `probes` are now `SampleSet` objects. Special care is taken to make sure the required attributes are defined for samples within them.
However there is no check for `SampleSet` attributes, which is not directly an issue. But later down the pipeline, when using checkpointed experiment, the `BioAlgCheckpointWrapper` rely on the `key` attribute to run (i.e. `_enroll_sample_set(...)`).Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/191Samples lose their metadata when going through PipelineSimple2023-04-04T17:16:21ZYannick DAYERSamples lose their metadata when going through PipelineSimpleSomewhere in the pipeline, the metadata of the references and probes (age, gender, ...) are lost. They are no longer available in the score files.
Issue encountered with `bob bio pipeline simple mobio-all arcface-insightface`.
After a ...Somewhere in the pipeline, the metadata of the references and probes (age, gender, ...) are lost. They are no longer available in the score files.
Issue encountered with `bob bio pipeline simple mobio-all arcface-insightface`.
After a first investigation:
- The metadata attributes are present when creating the `Sample` objects with the `Database`.
- The `ScoreWriter` receives `Sample` objects that are already missing those attributes.
- The `Distance` BioAlgorithm seems to transform samples into bags. Maybe more investigation needed here.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/46Dask Client configuration not available in installed package2023-03-28T12:52:29ZYannick DAYERDask Client configuration not available in installed packageWhen using the `bob bio simple` commands, the `dask.client` entry-points are not available.
Doing `bob bio pipeline simple -H conf.py` outputs in `conf.py`:
``` python
# ----------8<----------
# dask_client = single-threaded
"""Option...When using the `bob bio simple` commands, the `dask.client` entry-points are not available.
Doing `bob bio pipeline simple -H conf.py` outputs in `conf.py`:
``` python
# ----------8<----------
# dask_client = single-threaded
"""Optional parameter: dask_client (--dask-client, -l) [default: single-threaded]
Dask client for the execution of the pipeline. Can be a `dask.client' entry point, a module name, or a path to a Python file which contains a variable named `dask_client'.
Registered entries are: []"""
# ----------8<----------
```
Tried with the package installed from conda beta; also tried with `pip install -e`.
Entry points in bob.bio.base and bob.bio.face are working. So I presume it's an issue with how we do it in this package (maybe a wrong name for the entry-point group?).Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/96Annotation type XYZ is not supported.2023-03-08T10:00:15ZChristophe ECABERTAnnotation type XYZ is not supported.With recent changes to `CSVDataset`, running any baseline against `multipie` database is silently failing with the following message:
```bash
bob.bio.face.utils@2023-02-23 13:54:36,596 -- WARNING: Annotation type ('eyes-center', 'left-p...With recent changes to `CSVDataset`, running any baseline against `multipie` database is silently failing with the following message:
```bash
bob.bio.face.utils@2023-02-23 13:54:36,596 -- WARNING: Annotation type ('eyes-center', 'left-profile', 'right-profile') is not supported. Input images will be fully scaled.
```
There are two reasons for this behaviour:
- The change from `list` to `tuple` for the `MultipieDatabase.annotation_type` variable ([now](https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/master/src/bob/bio/face/database/multipie.py#L114), [then](https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/c8a6495ad85105661efa915e86a7eac7b1c2b3f6/bob/bio/face/database/multipie.py#L127))
- The oversimplified checks in [bob.bio.face.utils.py](https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/master/src/bob/bio/face/utils.py), checking only for list whereas it could potentially be any `Iterable` and not only `list`Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/95Set pytorch to run on single thread only on docker jobs2023-02-20T14:49:51ZYannick DAYERSet pytorch to run on single thread only on docker jobsTo fix the jobs on the docker runners for the CI, I added [this line](https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/d6d8e20bb73cfe4b099fedee603fad6498203d7f/src/bob/bio/face/embeddings/pytorch.py#L27) a while back.
We should:
- [x] r...To fix the jobs on the docker runners for the CI, I added [this line](https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/d6d8e20bb73cfe4b099fedee603fad6498203d7f/src/bob/bio/face/embeddings/pytorch.py#L27) a while back.
We should:
- [x] remove this line as this limits *any* work to a single thread and is not really wanted for performance reasons.
- [x] add `OMP_NUM_THREADS=1` as a variable in the CI config, either on the `.gitlab-ci.yaml` (in `bob/dev-profile`) (if possible, only for the jobs running on docker) or in the runner configuration (if possible?).André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/90Where went `CSVDatasetZTNorm` ?2023-02-20T08:45:24ZChristophe ECABERTWhere went `CSVDatasetZTNorm` ?With recent `csv-dataset` refactoring, some derived classes have been left on the side. With changes partially pushed to `bob` a simple import like `from bob.bio.face.database import MobioDatabase` will leads to:
```
ImportError: canno...With recent `csv-dataset` refactoring, some derived classes have been left on the side. With changes partially pushed to `bob` a simple import like `from bob.bio.face.database import MobioDatabase` will leads to:
```
ImportError: cannot import name 'CSVDatasetZTNorm' from 'bob.bio.base.database'
```
Which is quite inconvenient.https://gitlab.idiap.ch/bob/bob.bio.spear/-/issues/44Switch to new CI/CD configuration2023-02-14T12:19:19ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [ ] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.fusion.base/-/issues/12Switch to new CI/CD configuration2023-02-14T09:20:54ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.bio.base/-/issues/189Vulnerability analysis compares `template_id` instead of `subject_id`2023-02-13T16:35:57ZYannick DAYERVulnerability analysis compares `template_id` instead of `subject_id`To know if a score is genuine or an impostor, the fields to compare are `probe_subject_id` and `bio_ref_subject_id`.
The `vuln` commands currently use `probe_template_id` and `bio_ref_template_id`.To know if a score is genuine or an impostor, the fields to compare are `probe_subject_id` and `bio_ref_subject_id`.
The `vuln` commands currently use `probe_template_id` and `bio_ref_template_id`.Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.bio.vein/-/issues/28Switch to new CI/CD configuration2023-02-13T13:04:35ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/45CheckpointWrapper on annotator, saving the original dataset images as well as...2023-01-27T17:11:33ZAlain KOMATYCheckpointWrapper on annotator, saving the original dataset images as well as the annotations - waste of sapce!Hello,
When choosing to checkpoint in the pipeline, the annotator folder will contain the original images of the dataset instead of the annotations (face landmarks for example). One solution is the wrap a CheckpointWrapper around the an...Hello,
When choosing to checkpoint in the pipeline, the annotator folder will contain the original images of the dataset instead of the annotations (face landmarks for example). One solution is the wrap a CheckpointWrapper around the annotator. This will save the annotations in the annotator folder, but it will also save the original images, because now it is wrapped twice!
This problem comes from the [_wrap](https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/src/bob/pipelines/wrappers.py#L1014) function in the [wrappers](https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/src/bob/pipelines/wrappers.py) module.
Thanks to @cecabert, who pointed that in this function, there is no test whether the `estimator` is already an instance of CheckpointWrapper or not! One possible solution could be as follows (tested it and it is working for my pipelines):
```python
def _wrap(estimator, **kwargs):
# wrap the object and pass the kwargs
for w_class in bases:
valid_params = w_class._get_param_names()
params = {k: kwargs.pop(k) for k in valid_params if k in kwargs}
if estimator is None:
estimator = w_class(**params)
else:
if not isinstance(estimator, w_class):
estimator = w_class(estimator, **params)
return estimator, kwargs
```Yannick DAYERYannick DAYERhttps://gitlab.idiap.ch/bob/bob.pad.base/-/issues/44Switch to new CI/CD configuration2023-01-26T17:09:17ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZ