Can we assign K-Means and GMM M-steps to different queue?
Instead of using grid.training_queue for all kmeans-init, kmeans-m-step, gmm-init, and gmm-m-step, can we use say grid.enrollment_queue for kmeans-m-stepand gmm-m-step?
The reason I am suggesting this that initialization of KMeans and GMMs is usually much more demanding on memory (the features are loaded in memory) compared to m-steps (statistics are loaded). So, assigning all these steps to the same machine with large memory (which is usually a bottleneck resource), looks like a waste of resources and makes the whole training slow. Since, typically, grid.training_queue is memory demanding, while grid.enrollment_queue is not so much, it seem to make sense to assign these steps accordingly.
What do you think? @tiago.pereira @amohammadi @andre.anjos @mguenther