This issue was opened from the discussion on #38 (closed). In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.

To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:

• FPR -> False Positive Rate (spell it out so there are no confusions)
• FNR -> False Negative Rate
• Precision
• Recall
• F1-Score

And that is it

An option could allow you, for example, to replace the values above by something more digestible for biometrics, say --biometrics and then the program prints:

• False Acceptance Rate
• False Rejection Rate
• Half-Total Error Rate

The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":

• bob measure metrics dev-1.txt (as per above, use Equal-Error Rate to calculate the threshold, that should be also reported)
• bob measure metrics --biometrics dev-1.txt (as per above)
• bob measure metrics --far=0.0001 --biometrics dev-1.txt (reports values for an FAR of 0.0001 (0.01%), no minimisation takes place)
• bob measure metrics --criterion=minhter dev-1.txt (reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)

For obvious reasons, options such as --criterion and --far should be mutually exclusive. As it is currently coded, it is confusing that you should pass --criterion=far --far-value=0.0001. It would be easier to say --far=0.0001 and that is it. If the user passes both, then an error is raised.

It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.

Does that sound reasonable?