In a93954e1, I had introduced a test for the existence of files before evaluation -- such that the crash happens at the beginning of the evaluation and not in the middle. In that, I disregarded the --directory flag, which made this flag useless. Also, the tests for this flag were insufficient since the script was only tested with absolute paths.
This MR fixes the issue and updates the test case such that relative paths are passed to evaluate.py,