Hi @lcolbois,
Thanks for detecting the problem and opening the issue with all the details.
The problem is that most of the legacy algorithms are not pickled serializable, therefore, we can't wrap their transformations as :any:bob.pipelines.Sample and use pipelines in memory (vanilla-biometrics without the --checkpoint option).
Hence, we need to make them automatically "checkpointable" and use their legacy functions to persist data into the disk in a temporary directory.
This was wrongly set in the lgbphs baseline.
The /tmp/ was being set by default and this is not valid while running stuff on cluster.
Basically, you are persisting data on the /tmp of every node and obviously, this directory is not visible in other nodes.
That's why things work locally and on the grid with checkpoints.
In this MR bob.bio.base!238 (merged) I formalized a function that generates proper temporary directories that are visible on the cluster.
closes #39 (closed)