The verify.py script is very handy to trigger experiments. Not very recently @andre.anjos pushed a mechanism that allows us to provide configuration files as input and this made the work of preparing and triggering experiments even simpler and cleaner.

Imagine that your work consists of more than one hundred experiments with different databases (splits), preprocessors, feature extractors, etc. You can handle this complexity by spliting your configuration file in several parts and have something like this:

  $ verify.py config1.py config2.py config3.py

Although this makes the work clearner, this doesn't avoid you to call verify.py several times. A way to handle that is to have some sort of shell scripts that orchestrate these calls (or python scripts that generate the verify.py strings). Doesn't look very clean.

The script verify.py is not just a command line View for the experiment triggering; it handles the argument parsing, input validation and the experiment triggering. Today, it's not possible to detach the argument parsing from the input validation and the experiment triggering which makes impossible to programmatically trigger experiments.

We need some View-Controller design for this task.

For instance, in the View, we could wrap the content of the argument parsing in a dictionary (or use docopt that does this for free) and pass it to the Controller that will handle with the input validation and experiment triggering (it's just one way to handle this).

What would you think?

thanks

I'm willing to give a shot in this direction.