[VanillaBiometrics] Delayed Samples larger than memory

I guess @amohammadi is going to like this one.

IJB-C has an immense amount of probe samples (~128k) and summing up the size of all DelayedSamples, we have something around ~11GB of metadata. Remember that each probe sample holds the biometric reference keys that we should be comparing at scoring time.

If we run an experiment with IJB-C using our sge default setup that starts with one worker, (adaptive scale up/down workers due to load) vanilla-biometrics crashes in memory error before start doing work. Dask can't scale up the number of workers at this stage.

A solution for this is obviously to distribute this data among several workers. A possible way to approach this would be either, i-) to estimate the size of all DelayedSamples and create a heuristic to set n in dask_client.cluster.scale_up(n) before running .compute or ii-) add an optional argument in the vanilla biometrics CLI command allowing to set the number of workers to start vanilla-biometrics

ping @lcolbois

PS: XArrays, possibly in the future