Can we assign K-Means and GMM M-steps to different queue?
Instead of using grid.training_queue
for all kmeans-init
, kmeans-m-step
, gmm-init
, and gmm-m-step
, can we use say grid.enrollment_queue
for kmeans-m-step
and gmm-m-step
?
The reason I am suggesting this that initialization
of KMeans and GMMs is usually much more demanding on memory (the features are loaded in memory) compared to m-steps
(statistics are loaded). So, assigning all these steps to the same machine with large memory (which is usually a bottleneck resource), looks like a waste of resources and makes the whole training slow. Since, typically, grid.training_queue
is memory demanding, while grid.enrollment_queue
is not so much, it seem to make sense to assign these steps accordingly.
What do you think? @tiago.pereira @amohammadi @andre.anjos @mguenther