bob issueshttps://gitlab.idiap.ch/groups/bob/-/issues2023-01-25T08:30:14Zhttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/84[RFC] PyTorchModel2023-01-25T08:30:14ZChristophe ECABERT[RFC] PyTorchModelWith current implementation of the `PyTorchModel` the weights and the architecture need to be provided through `checkpoint_path` and `config` in order to use the transformer. This constraint can be alleviate using the `TorchScript` scrip...With current implementation of the `PyTorchModel` the weights and the architecture need to be provided through `checkpoint_path` and `config` in order to use the transformer. This constraint can be alleviate using the `TorchScript` script feature [ref](https://pytorch.org/docs/stable/jit.html).
TorchScript convert any `torch.nn.Module` into a persistent executable that can be loaded and used directly without needing to build the architecture first. It basically saves the "code" and the weights into a single file in the same fashion as Tensorflow. Moreover, some optimizations can be turned on during the saving phase such as converting all the ops into constant ops, freezing graph and so on.
Such mechanism can be used in the `PyTorchModel` base class to greatly simplify how we add new models. An example is provided below:
```python
class TorchScriptModel(TransformerMixin, BaseEstimator):
def __init__(self,
model_path,
preprocessor,
memory_demanding=False,
device=None,
**kwargs):
super().__init__(**kwargs)
self.memory_demanding = memory_demanding
# Model
self.model_path = model_path
self.model = None
self.preprocessor = preprocessor
if device is None:
device = pt.device('cpu')
self.device = device
def _model(self):
if self.model is None:
# For now, we suggest to disable the Jit Autocast Pass,
# As the issue: https://github.com/pytorch/pytorch/issues/75956
pt._C._jit_set_autocast_mode(False)
self.model = pt.jit.load(self.model_path)
self.model.eval()
self.model.to(self.device)
return self.model
def transform(self, X):
X = check_array(X, allow_nd=True).astype(np.float32)
model = self._model()
def _transform(x):
x = pt.from_numpy(x)
with pt.no_grad():
# Preprocess
x = x.to(self.device)
x = self.preprocessor(x)
# Extract embedding
x = model(x)
return x.cpu().numpy()
if self.memory_demanding:
features = []
for x in X:
f = _transform(x[None, ...])
features.append(f)
features = np.asarray(features)
if features.ndim >= 3:
features = np.vstack(features)
return features
else:
return _transform(X)
def __getstate__(self):
# Handling unpicklable objects
d = {}
for key, value in self.__dict__.items():
if key != 'model':
d[key] = value
d['model'] = None
return d
def _more_tags(self):
return {"requires_fit": False}
```
Moreover, some architecture are defined as a derived class of `PyTorchModel` such as `IResnetXXX` whereas others are defined through function returning a `PipelineSimple` instance. The consistency could be improved for instance by using `classmethod` to create a certain type of architecture. For instance it could be something like:
```python
class TorchScriptModel(TransformerMixin, BaseEstimator):
@classmethod
def IResNet(cls, version: Enum):
# Mechanic to retrive model from idiap server
return cls(...)
```
What are your thoughts on this @ydayer, @flavio.tarsetti, @lcolbois ?https://gitlab.idiap.ch/bob/bob.pad.face/-/issues/50Switch to new CI/CD configuration2023-01-05T09:26:49ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [ ] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.pipelines/-/issues/42Switch to new CI/CD configuration2023-01-02T14:21:40ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.bio.video/-/issues/24Switch to new CI/CD configuration2022-12-22T13:39:57ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Put the `requirements.txt` content inside `extra-intersphinx.txt`
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [ ] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.devtools/-/issues/112`bdt ci check` is executed on the nightlies job2022-12-22T10:28:13ZSamuel GAIST`bdt ci check` is executed on the nightlies jobWhy do the nightlies job run `bdt ci check` before doing their actual work ?
The way these checks are implemented are specific to bob's packages which the nightlies are not.
It breaks both beat and bob nigthlies run.Why do the nightlies job run `bdt ci check` before doing their actual work ?
The way these checks are implemented are specific to bob's packages which the nightlies are not.
It breaks both beat and bob nigthlies run.André AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.bio.face/-/issues/89Switch to new CI/CD configuration2022-12-16T17:33:32ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [ ] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/44check_parameters_for_validity does not always return the same type2022-12-06T11:13:39ZYannick DAYERcheck_parameters_for_validity does not always return the same typeCurrently, `bob.pipelines.utils.check_parameters_for_validity` can return ["a list or tuple"](https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/src/bob/pipelines/utils.py#L117).
This seems weird to return a list **or** a tuple. An...Currently, `bob.pipelines.utils.check_parameters_for_validity` can return ["a list or tuple"](https://gitlab.idiap.ch/bob/bob.pipelines/-/blob/master/src/bob/pipelines/utils.py#L117).
This seems weird to return a list **or** a tuple. And somewhere down the line, we actually expect a list (with a `remove` method).
Could you ensure that this returns a `list` in all cases (and edit the docstring to reflect that)?André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.bio.spear/-/issues/45Fix pipeline naming issue for GMM voxforge2022-12-05T16:15:08ZFlavio TARSETTIFix pipeline naming issue for GMM voxforgeAn issue was introduced when renaming pipelines configuration. The setup.py file seems to be corrupted with wrong pipeline information.An issue was introduced when renaming pipelines configuration. The setup.py file seems to be corrupted with wrong pipeline information.Roadmap to the major version of Bob 12Flavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/bob/bob.extension/-/issues/86Moving to Github and de-branding as a Bob package2022-11-23T08:31:30ZAndré AnjosMoving to Github and de-branding as a Bob packageThere is a general will to move software that can be used by a larger audience (that is not necessarily somebody at the @biometric group) to GitHub/conda-forge. This move would also de-brand this package as belonging to Bob.
To do this...There is a general will to move software that can be used by a larger audience (that is not necessarily somebody at the @biometric group) to GitHub/conda-forge. This move would also de-brand this package as belonging to Bob.
To do this, I propose we take on this task by first identifying the various bits in here that would be useful as standalone components. I find there are mainly 5 categories of functions:
- Build tools for C++: cmake, boost, pkgconfig, utils, __init__
- Helpers for Sphinx building: utils (`link_documentation`)
- Helpers to build CLIs: `scripts.click_helper`, `scripts.main_cli`
- Helpers for configuration: __init__, config, rc_config.py
- Helpers for logging: `log` (however some bits of it concern logging for C++)
I'm guessing that everything related to building other Bob packages (mostly the C++ code), can be considered deprecated once all C++ code has finally been ported to pure-Python alternatives. This then leaves us with the 4 other categories of helpers we have to somehow group (or not), to make packages.
Then, I propose we simply leave this package be (or archive it), and move the pieces of interest to a dedicated Python-only-builds GitHub project. We then ask each package going forward to make use of those specialised packages instead of bob.extension.https://gitlab.idiap.ch/bob/bob.devtools/-/issues/83[suggestion] Change CI template include style2022-11-17T17:35:27ZSamuel GAIST[suggestion] Change CI template include styleCurrently the `.gitlab-ci.yml` template for all bob packages contains either one or two lines like the following:
```
include: 'https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/data/gitlab-ci/single-package.yaml'
```
Th...Currently the `.gitlab-ci.yml` template for all bob packages contains either one or two lines like the following:
```
include: 'https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/data/gitlab-ci/single-package.yaml'
```
This has several drawbacks:
1) The server address is hardcoded
2) If there are incompatible changes to be done, there's no really easy way to test them without breaking a lot of packages
3) If one needs to keep using an old version of that file to be able to continue building it's not really clear how to proceed
For these reasons, I suggest to change the template as well as the bob packages `.gitlab-ci.yml` to the following.
```
include:
- project: 'bob/bob.devtools'
ref: master
file: '/bob/devtools/data/gitlab-ci/single-package.yaml'
```
This removes the server address so if there's a need to move stuff around, then there will be way less problems.
It also allows to retrieve the file from different branches if required in an easy fashion. For example if there's a massive change to the infrastructure, it allows to do that without breaking everything that is currently working.
The ref value can also be `$CI_DEFAULT_BRANCH` provided that the projects follow the same naming convention. It will thus allow to properly move from using master to main as default branch name.André AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.devtools/-/issues/110Overhauling our CI and pipelines2022-11-17T17:34:29ZAndré AnjosOverhauling our CI and pipelinesI'm proposing we should re-think the whole CI procedure entirely. Our CI system was made for the time we had some C++ code to be compiled and made available for other packages (@lcolbois: this was probably another reason for the splits)...I'm proposing we should re-think the whole CI procedure entirely. Our CI system was made for the time we had some C++ code to be compiled and made available for other packages (@lcolbois: this was probably another reason for the splits).
As of now and thanks to a lot of work by predecessors, most of the packages in this group are Python-only - this is certainly true for the (biometrics) nightlies. This means we can probably benefit from this fact and improve our pipelines, at least for most packages in the bob/beat namespace.
I have a proposal in mind, but I think we need to debate this a bit more thoroughly as I may not be thinking about all aspects and oversimplifying. The bulk of the proposal goes through revising our pipeline requirements and moving most of the packages into a python-only workflow (for most part). The proposed pipeline would be like this:
1. QA via `pre-commit`, if `.pre-commit-config.yaml` is present - only linux, should fail fast. Caching is enabled.
2. Test (via pytest) on various platforms and python versions - uses a Python docker image (not conda), respects python package pinning, retrieve beta versions from internal Gitlab package registry (if they exist), otherwise PyPI. Pip caching is enabled.
3. Generate sphinx documentation building and doctests (if `doc/` directory is present) - only linux, via Python docker image (not conda)
4. Package python code (via `python setup.py sdist`)
5. Package conda package using the sdist produced at the previous step, run tests (single platform - linux, just to cross-check)
6. Deploy sdist (unreleased beta) package on internal "group" package registry (GitLab: https://docs.gitlab.com/ee/user/packages/pypi_repository/) if not tagged or private/internal. Deploy sdist package on PyPI if tagged and public.
7. Deploy conda package on internal beta channel if not tagged, deploy on stable channel if tagged.
8. Deploy documentation and coverage information on DAV web server if success (master and stable, if required)
For non-python packages, maintain minimal CI configuration, but allow maximum flexibility via their own personalised `conda_build_config.yaml`. Everything else should match the above pipeline.
I also have some ideas on how to improve the packaging:
* Moving to `pyproject.toml` (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html), and
* Reflect that automatically on conda packages to deduplicate package lists (https://docs.conda.io/projects/conda-build/en/stable/resources/define-metadata.html#loading-data-from-other-files)
I expect this plan should make our pipelines run much faster, while simplifying the setup and testing. Conda environment creation continues to be supported and possible. For other simpler setups, simple Python software management (e.g. via pip or poetry) would be possible.
(This would affect issue #102 for example, #104 and maybe #103.)André AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.learn.em/-/issues/49dask problem with fitting GMMMachine ...2022-11-10T12:13:55ZPasra Rahimidask problem with fitting GMMMachine ...Can you guys check this error? The original script can be found at `/idiap/temp/prahimi/latentplay/gmm/gmm.py`
```bash
Loading latent data ...
Done loading.
(50000, 9216)
Working on GMMMachine with number of mixture models set to : 10...Can you guys check this error? The original script can be found at `/idiap/temp/prahimi/latentplay/gmm/gmm.py`
```bash
Loading latent data ...
Done loading.
(50000, 9216)
Working on GMMMachine with number of mixture models set to : 10
Traceback (most recent call last):
File "/somewhere/exps/latentplay/gmm/gmm.py", line 38, in <module>
machine = machine.fit(latents)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/bob/learn/em/gmm.py", line 792, in fit
self.initialize_gaussians(X)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/bob/learn/em/gmm.py", line 731, in initialize_gaussians
kmeans_machine = kmeans_machine.fit(data)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/bob/learn/em/kmeans.py", line 328, in fit
self.initialize(data=X)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/bob/learn/em/kmeans.py", line 312, in initialize
self.centroids_ = k_init(
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask_ml/cluster/k_means.py", line 365, in k_init
return init_scalable(X, n_clusters, random_state, max_iter, oversampling_factor)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask_ml/utils.py", line 550, in wraps
results = f(*args, **kwargs)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask_ml/cluster/k_means.py", line 423, in init_scalable
(cost,) = compute(evaluate_cost(X, centers))
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask_ml/cluster/k_means.py", line 486, in evaluate_cost
return (pairwise_distances(X, centers).min(1) ** 2).sum()
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask_ml/metrics/pairwise.py", line 59, in pairwise_distances
return X.map_blocks(
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask/array/core.py", line 2676, in map_blocks
return map_blocks(func, self, *args, **kwargs)
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask/array/core.py", line 873, in map_blocks
out = blockwise(
File "/somewhere/mambaforge/envs/latentplay/lib/python3.10/site-packages/dask/array/blockwise.py", line 269, in blockwise
raise ValueError(
ValueError: Dimension 1 has 2 blocks, adjust_chunks specified with 1 blocks
```Flavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/69Switch to new CI/CD configuration2022-11-10T10:14:15ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.bio.base/-/issues/188Switch to new CI/CD configuration2022-11-10T10:11:48ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [ ] Put the `requirements.txt` content inside `extra-intersphinx.txt`
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [ ] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.io.base/-/issues/24Switch to new CI/CD configuration2022-11-09T15:49:35ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.devtools/-/issues/104pyproject.toml not created in new packages2022-11-07T15:58:18ZYannick DAYERpyproject.toml not created in new packagesThe required `pyproject.toml` file is not in the project template and thus not created by the `bdt dev new ...` command.The required `pyproject.toml` file is not in the project template and thus not created by the `bdt dev new ...` command.https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/47Use the "src" folder layout2022-10-21T10:06:38ZYannick DAYERUse the "src" folder layoutWe should switch to the "src" layout and have all sources in an `src` folder, and tests in a `tests` folder.
It would be good to test `cookiecutter-pypackage` while doing this to have a clean base and see if it will be easy to use for t...We should switch to the "src" layout and have all sources in an `src` folder, and tests in a `tests` folder.
It would be good to test `cookiecutter-pypackage` while doing this to have a clean base and see if it will be easy to use for the upcoming bob packages upgrade.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/48Token for pypi deployment2022-10-19T16:24:39ZAndré MAYORAZToken for pypi deploymentThere is currently a problem with the local-pypi job that is supposed to deploy the package :
```
$ test -z "${CI_COMMIT_TAG}" && citool deregister --git-url "${CI_SERVER_URL}" --token "${PYPI_PACKAGE_REGISTRY_TOKEN}" --project "${CI_PRO...There is currently a problem with the local-pypi job that is supposed to deploy the package :
```
$ test -z "${CI_COMMIT_TAG}" && citool deregister --git-url "${CI_SERVER_URL}" --token "${PYPI_PACKAGE_REGISTRY_TOKEN}" --project "${CI_PROJECT_ID}" --name "${PACKAGE_NAME}" --version "${PACKAGE_VERSION}"
Traceback (most recent call last):
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/exceptions.py", line 333, in wrapped_f
return f(*args, **kwargs)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/mixins.py", line 246, in list
obj = self.gitlab.http_list(path, **data)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/client.py", line 942, in http_list
gl_list = GitlabList(self, url, query_data, get_next=False, **kwargs)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/client.py", line 1144, in __init__
self._query(url, query_data, **self._kwargs)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/client.py", line 1154, in _query
result = self._gl.http_request("get", url, query_data=query_data, **kwargs)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/client.py", line 798, in http_request
raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 403: 403 Forbidden
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/scratch/builds/bob/bob.learn.em/venv/bin/citool", line 8, in <module>
sys.exit(main())
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/citools/script.py", line 54, in main
args.func(args)
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/citools/deregister.py", line 43, in _wrapper
for p in proj.packages.list():
File "/scratch/builds/bob/bob.learn.em/venv/lib/python3.10/site-packages/gitlab/exceptions.py", line 335, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabListError: 403: 403 Forbidden
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
```
The problem is in the call of `for p in proj.packages.list():` in the `citools/deregister.py` file.
From what I investigated, the `gitlab.exceptions.GitlabListError: 403: 403 Forbidden` error could indicate that no token is provided to Gitlab, and therefore the runner cannot access to the packages.
For info, `proj.packages.list()` calls here:
```
curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.idiap.ch/api/v4/projects/1510/packages"
```
If an invalid token is given, it returns `{"message":"401 Unauthorized"}` but if no token is given, it returns `{"message":"403 Forbidden"}`
This would mean that `${PYPI_PACKAGE_REGISTRY_TOKEN}` is never set.
Where is it supposed to be set?
And why not use `${CI_JOB_TOKEN}` as `deregister.py` calls Gitlab API methods ?https://gitlab.idiap.ch/bob/bob.learn.em/-/issues/46Using ci-tools instead of bob devtools for the CI - process investigation2022-10-17T09:03:48ZFlavio TARSETTIUsing ci-tools instead of bob devtools for the CI - process investigationThis issue is to investigate the overall process of migrating a bob core package to `citools` instead of `bob.devtools`
The changes conducted to close this issue will help us understand what needs to be done for the other packages.This issue is to investigate the overall process of migrating a bob core package to `citools` instead of `bob.devtools`
The changes conducted to close this issue will help us understand what needs to be done for the other packages.Roadmap to the major version of Bob 12André MAYORAZAndré MAYORAZhttps://gitlab.idiap.ch/bob/bob.extension/-/issues/90Switch to new CI/CD configuration2022-10-12T14:12:44ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [ ] Modify `pyproject.toml`,
- Add information from `setup.py`,
- [ ] Empty `setup.py`,
- Leave the call to `setup()` for compatibility,
- [ ] Mo...We need to adapt this package to the new CI/CD and package format using citools:
- [ ] Modify `pyproject.toml`,
- Add information from `setup.py`,
- [ ] Empty `setup.py`,
- Leave the call to `setup()` for compatibility,
- [ ] Modify `conda/meta.yaml`,
- [ ] Remove test and documentation commands,
- [ ] Import names from `pyproject.toml`,
- [ ] Modify `.gitlab-ci.yml`,
- Use the nicer format instead of the URL,
- [ ] Add the project path to PATH in `doc/conf.py`,
- [ ] Remove `version.txt` and put the version in `pyproject.toml`.
You can look at bob.learn.em!68 for an example of a ported package.Roadmap to the major version of Bob 12