Trial to implement EM with multiprocessing
Currently, the EM by default runs in sequential mode, i.e., using only a single core of a multicore system. Implementing this using multithreading or multiprocessing would be great.
Here, I started implementing a version using multiprocessing
, where you can choose to take a multiprocessing.Pool
(for several processes) or a multiprocessing.pool.ThreadPool
for multiple threads.
Unfortunately it is not working yet, but @tiago.pereira might have an idea how to make it work.
Merge request reports
Activity
Reassigned to @tiago.pereira
Added enhancement label
@mguenther: would you have the time for the release by the end of the month for this one?
changed milestone to %Bob 2.7.x release
Not sure. Probably not. At least, I understand now, why it is not working at the moment. There are issues with pickling and unpickling C++ objects, which is required by
multiprocessing
to work. There would be solutions to implement a serialization in the C++ objects, but I am not sure if I would want to go down this path.Otherwise, we would need to implement to get the data of the C++ objects before sending them to the processes, and create new objects inside the
_parallel_e_step
function. I have successfully done that withLinearMachines
, e.g.:def _project(params): data, matrix, mean = params machine = bob.learn.linear.Machine(matrix) machine.input_subtract = mean return machine(data) def project_all(machine, all_data, pool): pool.map(_project, [(data, machine.weights, machine.mean) for data in all_data])
although this implementation is not very nice.
@tiago.pereira any news on this?
added 80 commits
-
64cc7198...709dd06b - 77 commits from branch
master
- ee4ca4aa - Trial to implement EM with multiprocessing
- b072592d - Doing EM with multiprocessing
- 1257136f - Merge branch 'multiprocessing' of gitlab.idiap.ch:bob/bob.learn.em into multiprocessing
Toggle commit list-
64cc7198...709dd06b - 77 commits from branch
It's working with multi-thread now.
I'm just looking how to release the GIL during the e-step.
ping @amohammadi
added 1 commit
- d8eb5517 - The data can be GMMStats not just numpy arrays
added 2 commits
- Resolved by Amir MOHAMMADI
added 1 commit
- b73834a6 - Automatically create threadpool and allow custom trainers
Hey @tiago.pereira I have tested this and pushed some minor modifications. Please review and merge it. LGTM.
- Resolved by Tiago de Freitas Pereira
33 34 Usually this list is generated by parallelizing the e-step of the ``trainer``. 34 35 """ 35 36 36 if isinstance(trainer, KMeansTrainer): 37 if trainer_type == "KMeansTrainer": 59 172 60 for i in range(max_iterations): 61 logger.debug("Iteration = %d/%d", i+1, max_iterations) 62 average_output_previous = average_output 63 trainer.m_step(machine, data) 64 trainer.e_step(machine, data) 173 if hasattr(trainer,"compute_likelihood"): 174 average_output = trainer.compute_likelihood(machine) 65 175 66 if hasattr(trainer, "compute_likelihood"): 67 average_output = trainer.compute_likelihood(machine) 176 for i in range(max_iterations): 177 logger.info("Iteration = %d/%d", i, max_iterations) 178 average_output_previous = average_output 179 trainer.m_step(machine, data) 180 _e_step(trainer, machine,data) @tiago.pereira, @andre.anjos do you know why do we run an E step after an M step? If this is the last iteration, looks like that the E step will go to waste, No?
pre-historical reasons.
I followed the design from
So, I could reuse the test cases as is.
mentioned in commit 1d35378c