Harmonisation of performance reporting

Hi @sebastien.marcel I have been working on this through my book chapter revision. I have been iterating on the design of this on #37 (closed) but if @heusch want to do it we should definitely coordinate.

See an example: plots.pdf

very good -- just talk to @heusch and @hmuckenhirn if they are still willing to help

Hi,

From the developer team, the original plan was to put @tgentilhomme on this task to get his feet wet on bob.

I propose a meeting tomorrow morning so I can present him to the biometrics team and then we can plan the tasks with him directly.

hi very good

go ahead meeting with the biometric team -- André and I are away unfortunately but we don't need to be here

however @heusch might be busy with a deadline

I let you check together

Then I propose tomorrow morning at 0900 in @heusch's office.

As mentioned by Seb, I'm quite busy: deadline on Thursday ... I would prefer on Friday morning then. Thanks

Sure, no problem.

@amohammadi Can you point @tgentilhomme to your work in progress on this matter ?

Hi the code is on my computer. It's not ready to be pushed yet. Do you want to work on this right away? I thought we had a meeting on Friday.

@amohammadi It would have been nice to have a better idea and understanding of what is done, what remains before Friday's meeting but it can wait if you prefer. Alternatively, I can drop by your office so that you can present my what you have done so far? Cheers.

Hi @tgentilhomme unfortunately I am sick today and I am working from home. I am trying to come up with a concrete plan of what needs to be done. In the meantime, you can look at #37 (closed) where I explain what I am planning to do. The issues that we have are referenced in #37 (closed). By reading the issues, you may be able to understand the problems that we have. You can also ask your technical questions in #37 (closed).

Ok I have come up with a clear set of tasks to get this going. This is by no means final though and is debatable.

Please see these issues to get context of the discussion: This issue #38 (closed) #37 (closed) bob.bio.base#60 (closed) https://gitlab.idiap.ch/bob/bob.pad.base/issues/1 bob.bio.base#65 (closed) #25 (closed) https://gitlab.idiap.ch/biometric/software/issues/9 https://gitlab.idiap.ch/biometric/software/issues/24

Tasks:

With the following considerations:

All plots should be according the ISO standards. Sebastien and Guillaume are the source for ISO informations.
All nan scores should be handled properly. Currently bob.measure ignores nan scores in weird ways.
All generic functionality must be in bob.measure. For example, bob.measure.roc and bob.measure.plot.roc will stay. Other missing ones like EPSC will be added to bob.measure. For example functions in https://gitlab.idiap.ch/bob/antispoofing.evaluation/blob/8d2fdd27846254bf517a5899b22fc9853ec5ea52/antispoofing/evaluation/utils/error_utils.py should go to bob.measure for EPSC plotting.
All scripts must be implemented under the general bob command using click: https://www.idiap.ch/software/bob/docs/bob/docs/master/bob.extension/doc/framework.html#unified-command-line-mechanism
All plotting functionality must be written in a modular and re-usable way. (e.g. they should not be click commands. click commands should call these functions)
The output of scripts should be capturable to a log file.

What sebastien wanted goes to:

bob bio evaluate FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
bob bio evaluate TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
bob bio evaluate nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
bob pad evaluate nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
bob pad evaluate ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
bob pad evaluate nice bar plot of score distributions for PAD (BonaFide, PA)
bob pad eps EPSC for biometric recognition and PAD
bob fusion scatter scatter plots for bi-modal biometric recognition
bob fusion scatter scatter plots for biometric recognition and PAD

You can see the initial implementation that I had in bob.pad.base!37 (closed) which I talked about in #37 (closed) However, since #39 (moved) was rejected. It's not possible to have #37 (closed) in bob.measure anymore.

ping @bob

Please feel free to edit my comment to keep the information here up-to-date (after we have discussed the change of course :) )

Hey guys, as discussed we thought it would be a good idea that you post some interesting plots and/or tables that you have produced before here so that we know what we are going to produce by the end of this. Please attach them as png images here so we can look at them without downloading them. For example here is an ROC image that I generated for my paper.

Development:

Evaluation:

There are two important details here.

The threshold is chosen on the development set (vertical line) and using the same threshold, evaluation points are drawn in the evaluation ROC figure.
The colors and line schemes are chosen in a way so that when you print this in grayscale, it's going to be readable still. (this something very specific to this plot though. I don't think this can and should be done for every figure).

Please post your interesting plots here.

Here is a couple more that I have:

BIO histogram:

PAD histogram:

Vulnerability histogram:

Educational Vulnerability histogram:

mentioned in issue #37 (closed)

Regarding the tasks #38 (comment 26050), I need your green light and inputs for the following:

#25 (closed) Move the biometric related functionality of bob.measure to bob.bio.base:

move bob.measure scripts (all of them?) to bob.bio.base scripts directory
move bob.measure load.py and openbr.py to bob.bio.base utils or tools (or somewhere else?)
move some (which ones?) of (or all?) the functions from __init__.py to a file (with what name?) in bob.bio.base (what directory?)? If some functions in __init__.py have to stay in bob.measure, wouldn't it better to move them in another file (what name?)? It is a little bit strange to have these kind of functions in __init__.py, no?

confidence interval calculations in bob.measure: https://gitlab.idiap.ch/bob/bob.paper.deepfacevuln2017/blob/e6969be231aaf6b34f1c1852c3623417379f5ca6/bob/paper/deepfacevuln2017/utils.py#L1.

Basically move it in bob.measure?

Regarding #38 (comment 26097), it would be great if you can give a list of all the plots/histo/tables you want, with an illustrative picture as suggested by @amohammadi and also test data or even a link to a potentially existing test function.

I will start ASAP.

Thank you

move bob.measure scripts (all of them?) to bob.bio.base scripts directory

No, please just remove them for now. We'll provide bob measure evaluate that does what these scripts does but using a different score file format.

move bob.measure load.py and openbr.py to bob.bio.base utils or tools (or somewhere else?)

Yes please create a folder called bob.bio.base.score for now.

move some (which ones?) of (or all?) the functions from __init__.py to a file (with what name?) in bob.bio.base (what directory?)? If some functions in __init__.py have to stay in bob.measure, wouldn't it better to move them in another file (what name?)? It is a little bit strange to have these kind of functions in __init__.py, no?

I don't think we are moving the functions from here. You need to ask this on #25 (closed).

Basically move it in bob.measure?

Yes with tests and documentation, please. You need to explain where this particular confidence interval can be used and where it cannot be used.

Regarding #38 (comment 26097), it would be great if you can give a list of all the plots/histo/tables you want, with an illustrative picture as suggested by @amohammadi and also test data or even a link to a potentially existing test function.

This is a bit hard to say now. Let's start with biometric plots that are implemented in evaluate.py in bob.bio.base. Sebastien gave a list of the plots too in this issue.

mentioned in merge request !52 (merged)

@theophile.gentilhomme's work on this is converging nicely. Because we wanted to release a version of Bob before merging his work, all his work is being merged into branches named theo. However, you should treat this merge requests as if they are being merged into master. So, if you want to have some say in them, please go ahead and review his merge requests (even the merged ones). After Bob is released we will just merge the theo branches into master (in bob.measure, bob.bio.base, and bob.pad.base).

Please see: https://gitlab.idiap.ch/groups/bob/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=theophile.gentilhomme for the list of merge requests.

Some example of plots. Let me know if this is okay with your expectations. Note that the commands come with numerous options that can change the display of the plots.

Bob measure commands

bob measure roc .../data/{dev,test}-{1,2}.txt -la 0.4,0.1

bob measure det .../data/{dev,test}-{1,2}.txt -la 0.4,0.1 -ts SysA,SysB

bob measure epc .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

bob measure hist .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

bob measure metrics .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

======  ===================  =============
SysA    Development dev-1    Eval. dev-2
======  ===================  =============
FtA     0.0%                 0.0%
FMR     6.3% (31/495)        6.2% (31/501)
FNMR    6.2% (28/451)        3.1% (14/446)
FAR     6.3%                 6.2%
FRR     6.2%                 3.1%
HTER    6.2%                 4.7%
======  ===================  =============
[Min. criterion: EER ] Threshold on Development set `SysB`: 1.652567e-02
======  ===================  ==============
SysB    Development dev-2    Eval. test-1
======  ===================  ==============
FtA     0.0%                 0.0%
FMR     4.6% (23/501)        5.0% (24/479)
FNMR    4.5% (20/446)        7.2% (34/473)
FAR     4.6%                 5.0%
FRR     4.5%                 7.2%
HTER    4.5%                 6.1%
======  ===================  ==============

Bob measure also comes with evaluate command that generates several plots and metrics at once and gen that generates fake scores for bob.measure supported input files.

These look gorgeous! Thanks for the work.

Harmonisation of performance reporting

Designs

Child items ...

Activity