Skip to content

Better handling of file structures with CheckpointWrapper

Some databases have file structures very flat (with more than 30k files in a directory). This is not good for the Idiap file structure and can let our I/O super slow.

We should implement a hash function in CheckpointWrapper.make_path that generates directory names given sample.key to limit the number of files in a directory to 1000 files.

ping @lcolbois (this touches experiments with IJB-C)