biopredict: Handle number of parallel jobs correctly
Merge request reports
Activity
added 32 commits
-
cb61f93a...a102df67 - 28 commits from branch
master
- d76cc9c2 - Handle number of parallel jobs correctly
- 54c52f1a - make reproducibility a function
- 517e0333 - rename test method
- caed652a - make tensorflow logging to warning
Toggle commit list-
cb61f93a...a102df67 - 28 commits from branch
- Resolved by Tiago de Freitas Pereira
31 32 # The below tf.set_random_seed() will make random number generation 33 # in the TensorFlow backend have a well-defined initial state. 34 # For further details, see: 35 # https://www.tensorflow.org/api_docs/python/tf/set_random_seed 36 tf_random_seed = 1234 37 tf.set_random_seed(tf_random_seed) 38 # sess = tf.Session(graph=tf.get_default_graph(), config=session_config) 39 # keras.backend.set_session(sess) 40 41 run_config = tf.estimator.RunConfig() 42 run_config = run_config.replace(session_config=session_config) 43 run_config = run_config.replace(tf_random_seed=tf_random_seed) 7 8 9 def set_seed(seed=0, python_hash_seed=0, log_device_placement=False): assigned to @tiago.pereira
assigned to @tiago.pereira
30 ----- 31 This functions return a list and its length might change. Please use 32 indices to select one of returned values. For example 33 ``sess_config, run_config = set_seed()[:2]``. 34 """ 35 # reproducible networks 36 # The below is necessary in Python 3.2.3 onwards to 37 # have reproducible behavior for certain hash-based operations. 38 # See these references for further details: 39 # https://docs.python.org/3.4/using/cmdline.html#envvar-PYTHONHASHSEED 40 # https://github.com/fchollet/keras/issues/2280#issuecomment-306959926 41 os.environ['PYTHONHASHSEED'] = '{}'.format(python_hash_seed) 42 43 # The below is necessary for starting Numpy generated random numbers 44 # in a well-defined initial state. 45 np.random.seed(seed) There's possible blunder here in case we want to continue the training in a future job.
tensorflow keeps track of the state of the pseudo random number generator using the global step as a reference, but for numpy we would need to do it explicitly by saving the current state
numpy.random.get_state
and loading the previous statenumpy.random.set_state
. I was doing this explicitly. We could do it too, everything is pickable.I don't have an automatic solution for it. Since we don't know when the job will stop, we should create a hook that saves the numpy random state a every X iterations, where X is in sync with
save_checkpoints_steps
.Then we create a function that is able to load this random state and trigger it in our
model_fn
s
mentioned in commit 0e818979
mentioned in issue #49