Skip to content

This package should only report, by default, generic metrics

This issue was opened from the discussion on #38 (closed). In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.

To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:

  • FPR -> False Positive Rate (spell it out so there are no confusions)
  • FNR -> False Negative Rate
  • Precision
  • Recall
  • F1-Score

And that is it

An option could allow you, for example, to replace the values above by something more digestible for biometrics, say --biometrics and then the program prints:

  • False Acceptance Rate
  • False Rejection Rate
  • Half-Total Error Rate

The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":

  • bob measure metrics dev-1.txt (as per above, use Equal-Error Rate to calculate the threshold, that should be also reported)
  • bob measure metrics --biometrics dev-1.txt (as per above)
  • bob measure metrics --far=0.0001 --biometrics dev-1.txt (reports values for an FAR of 0.0001 (0.01%), no minimisation takes place)
  • bob measure metrics --criterion=minhter dev-1.txt (reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)

For obvious reasons, options such as --criterion and --far should be mutually exclusive. As it is currently coded, it is confusing that you should pass --criterion=far --far-value=0.0001. It would be easier to say --far=0.0001 and that is it. If the user passes both, then an error is raised.

It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.

Does that sound reasonable?

Edited by André Anjos