Harmonisation of performance reporting
this is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn
currently we are going to get some support from the devel team led by Samuel @samuel.gaist so it will be good to synch with him as well
I have the impression that we are still a little bit behind with respect to the harmonisation of performance reporting as discussed before the Bob refactoring last year.
We are still reporting errors rates and plots with FRR/FAR/SFAR and EER and inconsistently FMR/FNMR/IAPRM/ACPER …
We should converge with an harmonisation following current practices that follow more and more ISO.
I know all the elements are in our hands (Tiago for CMCs, Amir for IAPMR and nice scatter plots with decision, …). See some examples attached.
We need a documented package with examples, on how to produce from a set of scores produced by our biometric and PAD experiments, that anyone can use to report results.
More particularly, we need to use
- FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
- TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
- nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
- nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
- ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
- nice bar plot of score distributions for PAD (BonaFide, PA)
- EPSC for biometric recognition and PAD
- scatter plots for bi-modal biometric recognition
- scatter plots for biometric recognition and PAD
Additionally we would need a routine to compute the statistical significance.
a summary of these performance reporting is provided in the attached document (section 4) prepared with our SWAN partners along with references to ISO documents (that can also be found in our biometrics group directory /idiap/group/biometric/standards/ISO-IEC/ eg. ISO-IEC-19795-1 ).
EPSC_HTER_w-cnn-motion-fusion.pdf
EPSC_IAPMR_w-cnn-motion-fusion.pdf
gmm_score_distribution_fixed.pdf
also a nice reference is to look at NIST FRVT ( https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt ) eg. http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=915761 and best practices document attached in interestingly in ROC/DET when they compare systems they draw lines between points with same threshold !
Sébastien