Skip to content
Snippets Groups Projects

biopredict: Handle number of parallel jobs correctly

Merged Amir MOHAMMADI requested to merge predict into master
2 unresolved threads

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • 31
    32 # The below tf.set_random_seed() will make random number generation
    33 # in the TensorFlow backend have a well-defined initial state.
    34 # For further details, see:
    35 # https://www.tensorflow.org/api_docs/python/tf/set_random_seed
    36 tf_random_seed = 1234
    37 tf.set_random_seed(tf_random_seed)
    38 # sess = tf.Session(graph=tf.get_default_graph(), config=session_config)
    39 # keras.backend.set_session(sess)
    40
    41 run_config = tf.estimator.RunConfig()
    42 run_config = run_config.replace(session_config=session_config)
    43 run_config = run_config.replace(tf_random_seed=tf_random_seed)
    7
    8
    9 def set_seed(seed=0, python_hash_seed=0, log_device_placement=False):
  • Amir MOHAMMADI removed assignee

    removed assignee

  • 30 -----
    31 This functions return a list and its length might change. Please use
    32 indices to select one of returned values. For example
    33 ``sess_config, run_config = set_seed()[:2]``.
    34 """
    35 # reproducible networks
    36 # The below is necessary in Python 3.2.3 onwards to
    37 # have reproducible behavior for certain hash-based operations.
    38 # See these references for further details:
    39 # https://docs.python.org/3.4/using/cmdline.html#envvar-PYTHONHASHSEED
    40 # https://github.com/fchollet/keras/issues/2280#issuecomment-306959926
    41 os.environ['PYTHONHASHSEED'] = '{}'.format(python_hash_seed)
    42
    43 # The below is necessary for starting Numpy generated random numbers
    44 # in a well-defined initial state.
    45 np.random.seed(seed)
    • There's possible blunder here in case we want to continue the training in a future job.

      tensorflow keeps track of the state of the pseudo random number generator using the global step as a reference, but for numpy we would need to do it explicitly by saving the current state numpy.random.get_state and loading the previous state numpy.random.set_state. I was doing this explicitly. We could do it too, everything is pickable.

    • I have had this problem too. How about python's random module?

    • It's the same. This module has also functions to get and set the state of the pseudo-random number generator.

    • If you have some better implementation in mind, please let me know then.

    • I don't have an automatic solution for it. Since we don't know when the job will stop, we should create a hook that saves the numpy random state a every X iterations, where X is in sync with save_checkpoints_steps.

      Then we create a function that is able to load this random state and trigger it in our model_fns

    • Please register or sign in to reply
  • mentioned in commit 0e818979

  • I will merge, we can handle this issue with numpy in the future

  • Amir MOHAMMADI mentioned in issue #49

    mentioned in issue #49

  • Please register or sign in to reply
    Loading