Skip to content

Plugin-based scripts for metrics and plots

we would like to add a script here:

$ bob measure

These scripts will load their subcommands from setuptools entrypoints; similar to how the bob ... loads its subcommands. This script will also help other packages to implement their evaluation scripts.

For example, we could have:

$ bob measure evaluate
# would print and plot according to FAR and FRR
$ bob bio evaluate
# would print and plot according to FMR and FNMR
$ bob pad evaluate
# would print and plot according to APCER and BPCER

We could also have commands that are specific to one plot or metric:

$ bob measure hter
$ bob measure hist
$ bob bio hist
$ bob pad hist
$ bob pad roc
$ bob face_icb2018 # would plot the figures of bob.paper.face_icb2018

It's desirable to be able to chain these commands:

$ bob measure hter hist roc det
$ bob bio hter hist roc

The original text of this issue is below

I would like to add two scripts here:

$ bob metrics
$ bob plots

These scripts will load their subcommands from setuptools entrypoints; similar to how the bob ... loads its subcommands.

Then, we can augment these scripts in bob.bio.base and bob.pad.base. For example, we could have:

$ bob metrics generic
# would print FAR and FRR
$ bob metrics bio
# would print FMR and FNMR
$ bob metrics pad
# would print APCER and BPCER

$ bob plots hist bio
# would plot histogram of genuines and zei
$ bob plots hist pad
# would plot histogram of bona-fide and PA
$ bob plots hist vuln
# would plot histogram of genuines, zei, and PA for vulnerability analysis

$ bob plots det bio
# would plot det curve with FMR and 1-FNMR

$ bob plots face_icb2018
# would plot the figures of bob.paper.face_icb2018

Any package could augment these with setuptools entrypoint and click while bob.measure could be a source of generic functions that other packages could take advantage of to easily implement these scripts.

What do you think?

I think this

  • Would encourage developers to create similar commands and re-use other's commands.
  • Would allow everybody to have its own plot script which easily discoverable by others.

I already have an implementation of this but it's in bob.pad.base. I think it would be better if it was in bob.measure.

I would like to possibly tackle other issues while doing this:

Edited by Amir MOHAMMADI