[VanillaBiometrics] Delayed Samples larger than memory
I guess @amohammadi is going to like this one.
IJB-C has an immense amount of probe samples (~128k) and summing up the size of all
DelayedSamples, we have something around ~11GB of metadata. Remember that each probe sample holds the biometric reference keys that we should be comparing at scoring time.
If we run an experiment with IJB-C using our
sge default setup that starts with one
worker, (adaptive scale up/down workers due to load) vanilla-biometrics crashes in memory error before start doing work.
Dask can't scale up the number of workers at this stage.
A solution for this is obviously to distribute this data among several workers.
A possible way to approach this would be either, i-) to estimate the size of all DelayedSamples and create a heuristic to set
dask_client.cluster.scale_up(n) before running
.compute or ii-) add an optional argument in the vanilla biometrics CLI command allowing to set the number of workers to start vanilla-biometrics
PS: XArrays, possibly in the future