[pyproject] Downgrade pytorch to avoid https://github.com/conda-forge/pytorch-cpu-feedstock/issues/254