[experiments] Reproducibility chart
After thinking about how reproducible an experiment in the platform really is, I think we can improve the experiment display a bit to include some sort of "reproducibility chart".
The idea behind this is to check, on a per-experiment basis and annotate, key points that may make the experiment irreproducible. Here are some key aspects
- The database used by the experiment is deactivated due to the end of the license agreement between Idiap and the controller
- The environment used by any of the blocks is not active anymore (outdated by another environment)
- There are incompatible API changes that may affect reproducibility (for example, the changes pushed on September 2nd. removing the
data_index
anddata_index_end
attributes). This condition must be taken with care, as it would require an (human) in-depth analysis of all experiment algorithms for a thorough conclusion. What we can do though, is to setup a new table that lists incompatible API changes and dates. If the experiment finished after that date, then we can consider it is compatible. Otherwise, not. We keep this table updated in case of eventual non-compatible API modifications.