Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • bob.measure bob.measure
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 5
    • Issues 5
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • bobbob
  • bob.measurebob.measure
  • Issues
  • #38
Closed
Open
Issue created Mar 02, 2018 by Sébastien MARCEL@sebastien.marcelOwner

Harmonisation of performance reporting

this is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn

currently we are going to get some support from the devel team led by Samuel @samuel.gaist so it will be good to synch with him as well


I have the impression that we are still a little bit behind with respect to the harmonisation of performance reporting as discussed before the Bob refactoring last year.

We are still reporting errors rates and plots with FRR/FAR/SFAR and EER and inconsistently FMR/FNMR/IAPRM/ACPER …

We should converge with an harmonisation following current practices that follow more and more ISO.

I know all the elements are in our hands (Tiago for CMCs, Amir for IAPMR and nice scatter plots with decision, …). See some examples attached.

We need a documented package with examples, on how to produce from a set of scores produced by our biometric and PAD experiments, that anyone can use to report results.

More particularly, we need to use

  • FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
  • TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
  • nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
  • nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
  • ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
  • nice bar plot of score distributions for PAD (BonaFide, PA)
  • EPSC for biometric recognition and PAD
  • scatter plots for bi-modal biometric recognition
  • scatter plots for biometric recognition and PAD

Additionally we would need a routine to compute the statistical significance.

a summary of these performance reporting is provided in the attached document (section 4) prepared with our SWAN partners along with references to ISO documents (that can also be found in our biometrics group directory /idiap/group/biometric/standards/ISO-IEC/ eg. ISO-IEC-19795-1 ).

EPSC_HTER_w-cnn-motion-fusion.pdf

EPSC_IAPMR_w-cnn-motion-fusion.pdf

gmm_score_distribution_fixed.pdf

ISV_gaussians.pdf

TR1-v3-20160930.pdf

also a nice reference is to look at NIST FRVT ( https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt ) eg. http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=915761 and best practices document attached in interestingly in ROC/DET when they compare systems they draw lines between points with same threshold !

060405-BestPractices_v2_1.pdf

Sébastien

Assignee
Assign to
Time tracking