Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • bob.bio.base bob.bio.base
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 15
    • Issues 15
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • bobbob
  • bob.bio.basebob.bio.base
  • Issues
  • #60
Closed
Open
Issue created Jan 26, 2017 by Amir MOHAMMADI@amohammadiOwner

Reporting the failed samples in the score files

According to the ISO/IEC 19795-1, several performance measures exist:

4.6.1 failure-to-enrol rate FTE
proportion of the population for whom the system fails to complete the enrolment process
NOTE The observed failure-to-enrol rate is measured on test crew enrolments. The predicted/expected failure-to-
enrol rate will apply to the entire target population.

4.6.2 failure-to-acquire rate FTA
proportion of verification or identification attempts for which the system fails to capture or locate an image or
signal of sufficient quality
NOTE The observed failure-to-acquire rate is distinct from the predicted/expected failure-to-acquire rate (the former
may be used to estimate the latter).

4.6.3 false non-match rate FNMR
proportion of genuine attempt samples falsely declared not to match the template of the same characteristic
from the same user supplying the sample
NOTE The measured/observed false non-match rate is distinct from the predicted/expected false non-match rate (the
former may be used to estimate the latter).

4.6.4 false match rate FMR
proportion of zero-effort impostor attempt samples falsely declared to match the compared non-self template
NOTE The measured/observed false match rate is distinct from the predicted/expected false match rate (the former
may be used to estimate the latter).

4.6.5 false reject rate FRR
proportion of verification transactions with truthful claims of identity that are incorrectly denied

4.6.6 false accept rate FAR
proportion of verification transactions with wrongful claims of identity that are incorrectly confirmed

And this is how to calculate them:

FRR = FTA + FNMR * (1 – FTA)
FAR = FMR * (1 – FTA)

However: "Comparison of systems having different failure-to-enrol rates may require use of generalized false reject (GFRR) and false accept rates (GFAR) which combine enrolment, sample acquisition and matching errors. The method of generalization should be appropriate to the evaluation."

Where one possible solution is: "A typical generalization is to treat a failure-to-enrol as if the enrolment completed, but all subsequent verification or identification transactions by that enrolee, or against their template, fail. The method of generalization shall be reported." which I think is good enough for us.

bob.bio.base handles failed samples with the --allow-missing-files option but the problem is normally you want to see traces of these failed samples in the score files too so that you can calculate the FTA. (edit these samples have NaN scores so you can calculate FTA)

I think the best way to do this would be to report numpy.nan in score files when something goes wrong.

Assignee
Assign to
Time tracking