Force setuptools=65 (downgrades pytorch-lightning as a result)

This MR touches upon the issue of upgrading setuptools (current version is 65.3.0), while keeping the current software stack. Unfortunately, tensorflow depends on protobuf 3.20.1 which blocks the whole update, forcing pytorch-lightning to be downgraded to an earlier version.

assigned to @andre.anjos

changed title from New pins to New conda-forge setup and pins

changed the description

added 1 commit

• 2992a8ee - [data/conda_build_config] Update pytorch-lightning

Compare with previous version

added 1 commit

• 1c5da4a2 - [data/conda_build_config] Update tensorboard

Compare with previous version

added 8 commits

• 1c5da4a2...fd0bc112 - 2 commits from branch master
• 2ee4fe2e - [data/conda_build_config] Fix compatibility with conda-forge
• ea2cc10b - [data/conda_build_config] Update pins
• 7f6509ee - [data/conda_build_config] Update pytorch-lightning
• c5f4b247 - [data/conda_build_config] Update tensorboard
• 65e1a647 - [bootstrap] update mamba version
• 5e2d2afd - [ci] Standardize caching

Compare with previous version

marked this merge request as draft

added 1 commit

• d55e7e48 - [data/conda_build_config] Pin setuptools first

Compare with previous version

In this proposal, we updated all packages, except pytorch-lightning, that is getting downgraded to 1.5.8 (from 1.6.4). I guess this package already has problems as reported by @lcolbois here: https://github.com/conda-forge/pytorch-lightning-feedstock/issues/93.

Running this: python bob/devtools/scripts/update_pins.py --python=3.10 setuptools=65 pytorch-lightning=1.7 does not work at all and requires setuptools to be set to 59.* so it actually can work. Probably due to the protobuf dependence.

added 1 commit

• 5f79a269 - [data/conda_build_config] Add cuda markers

Compare with previous version

added 1 commit

• b2b6b64e - [data/conda_build_config] Fix pytorch+cuda version

Compare with previous version

I should add that observed problems are due to tensorflow, which requires an incompatible version of protobuf. I'm not sure how many packages actually require tensorflow, or how fast that will happen. By removing tensorflow as a software stack dependence, we would be able to get everything updated correctly.

In the medium term we might have to decide if we want to keep handling several different DL libraries (Tensorflow, Pytorch, MXNet at least in bob.bio.face right now).

It would probably require significantly less effort to have a single one be the default (and I would guess PyTorch, given how I see things are evolving).

• PyTorch is becoming the standard in research
• Important open-source models made in other frameworks are typically quickly converted to PyTorch by the community
• Tools for conversion exist, e.g. ONNX (but disclaimer : I never used it and I don't know how mature it is), so we could convert ourselves remaining models. That's a once-only work that probably requires much less work than maintaining a multi-DL-libraries environment.

requested review from @lcolbois

I'm assigning this for @lcolbois to review. @bob: please note the downgrade of pytorch-lightning and the trade-off (remove tensorflow).

tensorflow is required by bob.bio.face at least. Newer protobuf support is about a few weeks away, for tensorflow: https://github.com/conda-forge/tensorflow-feedstock/pull/266

mentioned in merge request !327 (merged)

added 8 commits

• b2b6b64e...149f9a78 - 7 commits from branch master
• bb082961 - [data/conda_build_config] New pins that force setuptools=65

Compare with previous version

OK, I just force pushed the differences with respect to the conservative pins in !327 (merged). There are only 2 small changes here and we can see them clearly.

changed title from Draft: New conda-forge setup and pins to Draft: Force setuptools=65 (downgrades pytorch-lightning as a result)

changed the description