This package should only report, by default, generic metrics
This issue was opened from the discussion on #38 (closed). In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.
To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:
- FPR ->
False Positive Rate(spell it out so there are no confusions) - FNR ->
False Negative Rate - Precision
- Recall
- F1-Score
And that is it
An option could allow you, for example, to replace the values above by something more digestible for biometrics, say --biometrics and then the program prints:
- False Acceptance Rate
- False Rejection Rate
- Half-Total Error Rate
The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":
-
bob measure metrics dev-1.txt(as per above, use Equal-Error Rate to calculate the threshold, that should be also reported) -
bob measure metrics --biometrics dev-1.txt(as per above) -
bob measure metrics --far=0.0001 --biometrics dev-1.txt(reports values for an FAR of 0.0001 (0.01%), no minimisation takes place) -
bob measure metrics --criterion=minhter dev-1.txt(reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)
For obvious reasons, options such as --criterion and --far should be mutually exclusive. As it is currently coded, it is confusing that you should pass --criterion=far --far-value=0.0001. It would be easier to say --far=0.0001 and that is it. If the user passes both, then an error is raised.
It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.
Does that sound reasonable?