Skip to content
Snippets Groups Projects

Trial to implement EM with multiprocessing

Currently, the EM by default runs in sequential mode, i.e., using only a single core of a multicore system. Implementing this using multithreading or multiprocessing would be great.

Here, I started implementing a version using multiprocessing, where you can choose to take a multiprocessing.Pool (for several processes) or a multiprocessing.pool.ThreadPool for multiple threads. Unfortunately it is not working yet, but @tiago.pereira might have an idea how to make it work.

Edited by Tiago de Freitas Pereira

Merge request reports

Pipeline #30022 passed

Pipeline passed for 510bd6ab on multiprocessing

Test coverage 91.00% from 2 jobs
Approval is optional

Merged by Tiago de Freitas PereiraTiago de Freitas Pereira May 14, 2019 (May 14, 2019 11:26am UTC)

Merge details

  • Changes merged into master with 1d35378c.
  • Deleted the source branch.

Pipeline #30161 passed

Pipeline passed for 1d35378c on master

Test coverage 91.00% from 2 jobs

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • added 1 commit

    Compare with previous version

  • Amir MOHAMMADI added 1 commit

    added 1 commit

    • b73834a6 - Automatically create threadpool and allow custom trainers

    Compare with previous version

  • Amir MOHAMMADI added 1 commit

    added 1 commit

    Compare with previous version

  • Amir MOHAMMADI resolved all discussions

    resolved all discussions

  • Hey @tiago.pereira I have tested this and pushed some minor modifications. Please review and merge it. LGTM.

  • Amir MOHAMMADI added 1 commit

    added 1 commit

    Compare with previous version

  • Awesome. Thanks

    I need to push some modifications just to get it nicer :-)

  • Tiago de Freitas Pereira resolved all discussions

    resolved all discussions

  • 33 34 Usually this list is generated by parallelizing the e-step of the ``trainer``.
    34 35 """
    35 36
    36 if isinstance(trainer, KMeansTrainer):
    37 if trainer_type == "KMeansTrainer":
  • added 1 commit

    Compare with previous version

  • Hi, I just pushed the TV training multithread.

  • 59 172
    60 for i in range(max_iterations):
    61 logger.debug("Iteration = %d/%d", i+1, max_iterations)
    62 average_output_previous = average_output
    63 trainer.m_step(machine, data)
    64 trainer.e_step(machine, data)
    173 if hasattr(trainer,"compute_likelihood"):
    174 average_output = trainer.compute_likelihood(machine)
    65 175
    66 if hasattr(trainer, "compute_likelihood"):
    67 average_output = trainer.compute_likelihood(machine)
    176 for i in range(max_iterations):
    177 logger.info("Iteration = %d/%d", i, max_iterations)
    178 average_output_previous = average_output
    179 trainer.m_step(machine, data)
    180 _e_step(trainer, machine,data)
  • mentioned in commit 1d35378c

  • Please register or sign in to reply
    Loading