generic plotting script for bob measure

From #37 (closed), provide generic plotting scripts:

• bob measure evaluate
• bob measure hist
• bob measure hter
• ...

@amohammadi @mguenther @andre.anjos @tiago.pereira Could you please let me know what figures should bob measure evaluate plot? So far, it has the exact same outputs than compute_perf.

Same question for bob measure hist and possibly other specific commands (please provide a list). It would be nice to have a reference plot (image or reference to a paper/report) for each one as in #38 (comment 26097), and possibly a test dataset.

@theophile.gentilhomme I would like to see better use of click functionality here. Especially plot_options and normalize_options are not a good way to go forward, I think. Maybe we can have decorators for each parameter and their callback also inside the decorator? similar to https://gitlab.idiap.ch/bob/bob.extension/blob/d2f1140236fbd3e19fc37897c0c252067ca5c4cb/bob/extension/scripts/click_helper.py#L13

also multi-command chaining: http://click.pocoo.org/6/commands/#multi-command-chaining

You need to also use click.echo instead of print.

I think for now whatever compute_perf had is good for a list of plots for bob.measure. We would be able to discuss and iterate this better when you are implementing bob bio evaluate.

Hi @amohammadi . Thanks for your comments. This is in no way the final implementation. Just starting point for discussions. I am not using plot_options and normalize_options so I can remove them. Cheers.

added 1 commit

• cb12cf37 - More input tests

Compare with previous version

@amohammadi Not sure how you want to implement multi-command chaining. What I understood is that you want something like bob measure roc det hit .../path/to/scores/dev-score .../path/to/scores/test-score . From my understanding it is not possible (or very difficult) to do it with Click as I do not see how to share the last sub-command arguments with other commands.

One way to use chaining if to input the arguments after the measure command and pass to the sub-command, but it can only take a limited number of arguments.

What I plan to do is implement individual commands for roc, det, hist, ... and evaluate. In evaluate, it might be possible to call other commands as described in http://click.pocoo.org/5/advanced/#invoking-other-commands which would avoid code duplication (but might be less efficient if all the commands need to load inputs, etc...). As @amohammadi said #37 (comment 26294), in this case it would be equivalent to bob measure roc det hist ..../scores.

Please let me know what you think.

Please go ahead. It sounds reasonable.

Could you please let me know what figures should bob measure evaluate plot? So far, it has the exact same outputs than compute_perf?

I think, a score distribution plot would be nice. I think, this plot would be generic enough to be inside bob.measure. I believe that we do not have such a plot yet, so you'd need to design and write the code for it.

mentioned in merge request bob.bio.base!138 (merged)

added 1 commit

• c9789352 - Add measure scripts: metrics (HTER, EER), roc, det, epc, hist (score histograms)…

Compare with previous version

added 1 commit

• 2131bb62 - Delete test file

Compare with previous version

added 1 commit

• 2b342577 - Change test data

Compare with previous version

added 1 commit

• 09e634bc - Missing requirements

Compare with previous version

added 1 commit

• a62f5710 - Requirement fix

Compare with previous version

CI does not manage to install the package tabulate with error:

Undefined Jinja2 variables remain (['tabulate']). Please enable source downloading and try again.

Note: The .../conda/meta.yaml and requirements.txt have been modified to add this dependency.

@andre.anjos @amohammadi Any idea how to fix this? Cheers.

All package dependencies must be tracked so we can create a conda-installable software stack. Direct installation from PyPI would break this requirement and are therefore forbidden.

Here is a recipe to get you going:

1. Check if the package you want to use is conda-installable from the defaults channel via conda search <name>
2. If that is not the case, you'll have to package and provide this dependence in bob/bob.conda
3. Once the dependence is available for installation, either via defaults or our own Idiap channel, proceed and:
   1. Make sure you have the dependence listed on the file requirements.txt in this package
   2. Make sure you have the dependence listed on the conda/meta.yaml relevant section

Once all this is in place, you can use the new dependence on your package.

Caution must be taken when adding dependencies to a package we maintain:

1. Prefer packages that already have a conda package in the defaults channel (that reduces maintenance work)
2. If that is not possible, avoid unmaintained or brand new packages as that may stop working at any time (needless to say, you'll have to "support" these)
3. Prefer packages which have a long-time history and maintenance schedule

Hi @andre.anjos. Thanks for your reply.

Check if the package you want to use is conda-installable from the defaults channel via conda search

tabulate is a default package

$ conda search tabulate Loading channels: done Name Version Build Channel
tabulate 0.7.5 py27_0 defaults
tabulate 0.7.5 py34_0 defaults
tabulate 0.7.5 py35_0 defaults
tabulate 0.8.2 py27_0 defaults
tabulate 0.8.2 py35_0 defaults
tabulate 0.8.2 py36_0 defaults

as displayed above.

Make sure you have the dependence listed on the file requirements.txt in this package

Yes it was. https://gitlab.idiap.ch/bob/bob.measure/blob/theo-cli/requirements.txt

Make sure you have the dependence listed on the conda/meta.yaml relevant section

Yes it was. https://gitlab.idiap.ch/bob/bob.measure/blob/theo-cli/conda/meta.yaml

But I still get this error. Cheers.

The issue here is that tabulate is not included in bob/bob.admin/gitlab/conda_build_config.yaml and, therefore, conda cannot resolve the { tabulate } variable you're mentioning on your recipe.

The way to fix it is to insert this dependence on our software stack via separate commits: one to bob/bob.conda/conda/bob-devel/meta.yaml and the second via the config file mentioned above. @tiago.pereira is on it at bob/bob.conda!344. More later.

@theophile.gentilhomme: once that is resolved, @tiago.pereira shall give you a green light.

Hey @theophile.gentilhomme. It's done.

I'll trigger the pipeline if you don't mind.

Cheers

Thanks @tiago.pereira .