Reporting the failed samples in the score files
According to the ISO/IEC 19795-1, several performance measures exist:
4.6.1 failure-to-enrol rate FTE proportion of the population for whom the system fails to complete the enrolment process NOTE The observed failure-to-enrol rate is measured on test crew enrolments. The predicted/expected failure-to- enrol rate will apply to the entire target population. 4.6.2 failure-to-acquire rate FTA proportion of verification or identification attempts for which the system fails to capture or locate an image or signal of sufficient quality NOTE The observed failure-to-acquire rate is distinct from the predicted/expected failure-to-acquire rate (the former may be used to estimate the latter). 4.6.3 false non-match rate FNMR proportion of genuine attempt samples falsely declared not to match the template of the same characteristic from the same user supplying the sample NOTE The measured/observed false non-match rate is distinct from the predicted/expected false non-match rate (the former may be used to estimate the latter). 4.6.4 false match rate FMR proportion of zero-effort impostor attempt samples falsely declared to match the compared non-self template NOTE The measured/observed false match rate is distinct from the predicted/expected false match rate (the former may be used to estimate the latter). 4.6.5 false reject rate FRR proportion of verification transactions with truthful claims of identity that are incorrectly denied 4.6.6 false accept rate FAR proportion of verification transactions with wrongful claims of identity that are incorrectly confirmed
And this is how to calculate them:
FRR = FTA + FNMR * (1 – FTA) FAR = FMR * (1 – FTA)
However: "Comparison of systems having different failure-to-enrol rates may require use of generalized false reject (GFRR) and false accept rates (GFAR) which combine enrolment, sample acquisition and matching errors. The method of generalization should be appropriate to the evaluation."
Where one possible solution is: "A typical generalization is to treat a failure-to-enrol as if the enrolment completed, but all subsequent verification or identification transactions by that enrolee, or against their template, fail. The method of generalization shall be reported." which I think is good enough for us.
bob.bio.base handles failed samples with the
--allow-missing-files option but the problem is normally you want to see traces of these failed samples in the score files too so that you can calculate the FTA. (edit these samples have NaN scores so you can calculate FTA)
I think the best way to do this would be to report
numpy.nan in score files when something goes wrong.