Reporting the failed samples in the score files

According to the ISO/IEC 19795-1, several performance measures exist:

4.6.1 failure-to-enrol rate FTE
proportion of the population for whom the system fails to complete the enrolment process
NOTE The observed failure-to-enrol rate is measured on test crew enrolments. The predicted/expected failure-to-
enrol rate will apply to the entire target population.

4.6.2 failure-to-acquire rate FTA
proportion of verification or identification attempts for which the system fails to capture or locate an image or
signal of sufficient quality
NOTE The observed failure-to-acquire rate is distinct from the predicted/expected failure-to-acquire rate (the former
may be used to estimate the latter).

4.6.3 false non-match rate FNMR
proportion of genuine attempt samples falsely declared not to match the template of the same characteristic
from the same user supplying the sample
NOTE The measured/observed false non-match rate is distinct from the predicted/expected false non-match rate (the
former may be used to estimate the latter).

4.6.4 false match rate FMR
proportion of zero-effort impostor attempt samples falsely declared to match the compared non-self template
NOTE The measured/observed false match rate is distinct from the predicted/expected false match rate (the former
may be used to estimate the latter).

4.6.5 false reject rate FRR
proportion of verification transactions with truthful claims of identity that are incorrectly denied

4.6.6 false accept rate FAR
proportion of verification transactions with wrongful claims of identity that are incorrectly confirmed

And this is how to calculate them:

FRR = FTA + FNMR * (1 – FTA)
FAR = FMR * (1 – FTA)

However: "Comparison of systems having different failure-to-enrol rates may require use of generalized false reject (GFRR) and false accept rates (GFAR) which combine enrolment, sample acquisition and matching errors. The method of generalization should be appropriate to the evaluation."

Where one possible solution is: "A typical generalization is to treat a failure-to-enrol as if the enrolment completed, but all subsequent verification or identification transactions by that enrolee, or against their template, fail. The method of generalization shall be reported." which I think is good enough for us.

bob.bio.base handles failed samples with the --allow-missing-files option but the problem is normally you want to see traces of these failed samples in the score files too so that you can calculate the FTA. (edit these samples have NaN scores so you can calculate FTA)

I think the best way to do this would be to report numpy.nan in score files when something goes wrong.