Harmonisation of performance reporting

Hi @sebastien.marcel I have been working on this through my book chapter revision. I have been iterating on the design of this on #37 (closed) but if @heusch want to do it we should definitely coordinate.

See an example: plots.pdf

very good -- just talk to @heusch and @hmuckenhirn if they are still willing to help

Hi,

From the developer team, the original plan was to put @tgentilhomme on this task to get his feet wet on bob.

I propose a meeting tomorrow morning so I can present him to the biometrics team and then we can plan the tasks with him directly.

hi very good

go ahead meeting with the biometric team -- André and I are away unfortunately but we don't need to be here

however @heusch might be busy with a deadline

I let you check together

Then I propose tomorrow morning at 0900 in @heusch's office.

As mentioned by Seb, I'm quite busy: deadline on Thursday ... I would prefer on Friday morning then. Thanks

Sure, no problem.

@amohammadi Can you point @tgentilhomme to your work in progress on this matter ?

Hi the code is on my computer. It's not ready to be pushed yet. Do you want to work on this right away? I thought we had a meeting on Friday.

@amohammadi It would have been nice to have a better idea and understanding of what is done, what remains before Friday's meeting but it can wait if you prefer. Alternatively, I can drop by your office so that you can present my what you have done so far? Cheers.

Hi @tgentilhomme unfortunately I am sick today and I am working from home. I am trying to come up with a concrete plan of what needs to be done. In the meantime, you can look at #37 (closed) where I explain what I am planning to do. The issues that we have are referenced in #37 (closed). By reading the issues, you may be able to understand the problems that we have. You can also ask your technical questions in #37 (closed).

Ok I have come up with a clear set of tasks to get this going. This is by no means final though and is debatable.

Please see these issues to get context of the discussion: This issue #38 (closed) #37 (closed) bob.bio.base#60 (closed) https://gitlab.idiap.ch/bob/bob.pad.base/issues/1 bob.bio.base#65 (closed) #25 (closed) https://gitlab.idiap.ch/biometric/software/issues/9 https://gitlab.idiap.ch/biometric/software/issues/24

Tasks:

With the following considerations:

All plots should be according the ISO standards. Sebastien and Guillaume are the source for ISO informations.
All nan scores should be handled properly. Currently bob.measure ignores nan scores in weird ways.
All generic functionality must be in bob.measure. For example, bob.measure.roc and bob.measure.plot.roc will stay. Other missing ones like EPSC will be added to bob.measure. For example functions in https://gitlab.idiap.ch/bob/antispoofing.evaluation/blob/8d2fdd27846254bf517a5899b22fc9853ec5ea52/antispoofing/evaluation/utils/error_utils.py should go to bob.measure for EPSC plotting.
All scripts must be implemented under the general bob command using click: https://www.idiap.ch/software/bob/docs/bob/docs/master/bob.extension/doc/framework.html#unified-command-line-mechanism
All plotting functionality must be written in a modular and re-usable way. (e.g. they should not be click commands. click commands should call these functions)
The output of scripts should be capturable to a log file.

What sebastien wanted goes to:

bob bio evaluate FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
bob bio evaluate TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
bob bio evaluate nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
bob pad evaluate nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
bob pad evaluate ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
bob pad evaluate nice bar plot of score distributions for PAD (BonaFide, PA)
bob pad eps EPSC for biometric recognition and PAD
bob fusion scatter scatter plots for bi-modal biometric recognition
bob fusion scatter scatter plots for biometric recognition and PAD

You can see the initial implementation that I had in bob.pad.base!37 (closed) which I talked about in #37 (closed) However, since #39 (moved) was rejected. It's not possible to have #37 (closed) in bob.measure anymore.

ping @bob

Please feel free to edit my comment to keep the information here up-to-date (after we have discussed the change of course :) )

Hey guys, as discussed we thought it would be a good idea that you post some interesting plots and/or tables that you have produced before here so that we know what we are going to produce by the end of this. Please attach them as png images here so we can look at them without downloading them. For example here is an ROC image that I generated for my paper.

Development:

Evaluation:

There are two important details here.

The threshold is chosen on the development set (vertical line) and using the same threshold, evaluation points are drawn in the evaluation ROC figure.
The colors and line schemes are chosen in a way so that when you print this in grayscale, it's going to be readable still. (this something very specific to this plot though. I don't think this can and should be done for every figure).

Please post your interesting plots here.

Here is a couple more that I have:

BIO histogram:

PAD histogram:

Vulnerability histogram:

Educational Vulnerability histogram:

mentioned in issue #37 (closed)

Regarding the tasks #38 (comment 26050), I need your green light and inputs for the following:

#25 (closed) Move the biometric related functionality of bob.measure to bob.bio.base:

move bob.measure scripts (all of them?) to bob.bio.base scripts directory
move bob.measure load.py and openbr.py to bob.bio.base utils or tools (or somewhere else?)
move some (which ones?) of (or all?) the functions from __init__.py to a file (with what name?) in bob.bio.base (what directory?)? If some functions in __init__.py have to stay in bob.measure, wouldn't it better to move them in another file (what name?)? It is a little bit strange to have these kind of functions in __init__.py, no?

confidence interval calculations in bob.measure: https://gitlab.idiap.ch/bob/bob.paper.deepfacevuln2017/blob/e6969be231aaf6b34f1c1852c3623417379f5ca6/bob/paper/deepfacevuln2017/utils.py#L1.

Basically move it in bob.measure?

Regarding #38 (comment 26097), it would be great if you can give a list of all the plots/histo/tables you want, with an illustrative picture as suggested by @amohammadi and also test data or even a link to a potentially existing test function.

I will start ASAP.

Thank you

move bob.measure scripts (all of them?) to bob.bio.base scripts directory

No, please just remove them for now. We'll provide bob measure evaluate that does what these scripts does but using a different score file format.

move bob.measure load.py and openbr.py to bob.bio.base utils or tools (or somewhere else?)

Yes please create a folder called bob.bio.base.score for now.

move some (which ones?) of (or all?) the functions from __init__.py to a file (with what name?) in bob.bio.base (what directory?)? If some functions in __init__.py have to stay in bob.measure, wouldn't it better to move them in another file (what name?)? It is a little bit strange to have these kind of functions in __init__.py, no?

I don't think we are moving the functions from here. You need to ask this on #25 (closed).

Basically move it in bob.measure?

Yes with tests and documentation, please. You need to explain where this particular confidence interval can be used and where it cannot be used.

Regarding #38 (comment 26097), it would be great if you can give a list of all the plots/histo/tables you want, with an illustrative picture as suggested by @amohammadi and also test data or even a link to a potentially existing test function.

This is a bit hard to say now. Let's start with biometric plots that are implemented in evaluate.py in bob.bio.base. Sebastien gave a list of the plots too in this issue.

mentioned in merge request !52 (merged)

@theophile.gentilhomme's work on this is converging nicely. Because we wanted to release a version of Bob before merging his work, all his work is being merged into branches named theo. However, you should treat this merge requests as if they are being merged into master. So, if you want to have some say in them, please go ahead and review his merge requests (even the merged ones). After Bob is released we will just merge the theo branches into master (in bob.measure, bob.bio.base, and bob.pad.base).

Please see: https://gitlab.idiap.ch/groups/bob/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=theophile.gentilhomme for the list of merge requests.

Some example of plots. Let me know if this is okay with your expectations. Note that the commands come with numerous options that can change the display of the plots.

Bob measure commands

bob measure roc .../data/{dev,test}-{1,2}.txt -la 0.4,0.1

bob measure det .../data/{dev,test}-{1,2}.txt -la 0.4,0.1 -ts SysA,SysB

bob measure epc .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

bob measure hist .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

bob measure metrics .../data/{dev,test}-{1,2}.txt -ts SysA,SysB

======  ===================  =============
SysA    Development dev-1    Eval. dev-2
======  ===================  =============
FtA     0.0%                 0.0%
FMR     6.3% (31/495)        6.2% (31/501)
FNMR    6.2% (28/451)        3.1% (14/446)
FAR     6.3%                 6.2%
FRR     6.2%                 3.1%
HTER    6.2%                 4.7%
======  ===================  =============
[Min. criterion: EER ] Threshold on Development set `SysB`: 1.652567e-02
======  ===================  ==============
SysB    Development dev-2    Eval. test-1
======  ===================  ==============
FtA     0.0%                 0.0%
FMR     4.6% (23/501)        5.0% (24/479)
FNMR    4.5% (20/446)        7.2% (34/473)
FAR     4.6%                 5.0%
FRR     4.5%                 7.2%
HTER    4.5%                 6.1%
======  ===================  ==============

Bob measure also comes with evaluate command that generates several plots and metrics at once and gen that generates fake scores for bob.measure supported input files.

These look gorgeous! Thanks for the work.

mentioned in issue bob.bio.base#120 (closed)

@theophile.gentilhomme

I have encountered an issue with bob bio evaluate and bob measure metrics commands and I hope this is a correct place to report it.

bob bio evaluate: when given two score files, dev and eval, it computes EER and MIN-HTER thresholds independently on each file. However, it should compute a threshold on dev scores and apply the same threshold on eval scores in each EER and MIN-HTER scenarios.
bob measure metrics: I thought may be the other command would work. I gave the same dev and eval score files to this command and I got an Error telling me that the file must by a two column file with the first column containing -1 or 1 and the second column the scores. I am a bit confused. Do we use two column files anywhere?

@pkorshunov

Please use --eval option when you pass evaluation score along with dev scores
Please see !50 (merged), #39 (comment 26366)

I just noticed that this package still reports errors following biometrics naming conventions. IMO, biometrics-related stuff (e.g., the distinction between FAR FMR) should be moved up to bob.bio.base. Here, you should have things that look more like https://en.wikipedia.org/wiki/Precision_and_recall.

Let make things square before changing the names once again:

@andre.anjos : You'are talking about bob measure metrics outputs, right?

If so you want:

Remove fta
Remove FMR FMNR
Rename FAR to something like FPR (False positive rate).
Rename FRR to something like FNR
What about HTER ? Is it neutral enough?

Let me know if this is correct and the name is propose are OK. Otherwise, please give me more precision and propose names for those quantities.

ping @amohammadi

Cheers

To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:

FPR -> False Positive Rate (spell it out so there are no confusions)
FNR -> False Negative Rate
Precision
Recall
F1-Score

And that is it

An option could allow you, for example, to replace the values above by something more digestible for biometrics, say --biometrics and then the program prints:

False Acceptance Rate
False Rejection Rate
Half-Total Error Rate

The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":

bob measure metrics dev-1.txt (as per above, use Equal-Error Rate to calculate the threshold, that should be also reported)
bob measure metrics --biometrics dev-1.txt (as per above)
bob measure metrics --far=0.0001 --biometrics dev-1.txt (reports values for an FAR of 0.0001 (0.01%), no minimisation takes place)
bob measure metrics --criterion=minhter dev-1.txt (reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)

For obvious reasons, options such as --criterion and --far should be mutually exclusive. As it is currently coded, it is confusing that you should pass --criterion=far --far-value=0.0001. It would be easier to say --far=0.0001 and that is it. If the user passes both, then an error is raised.

It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.

Does that sound reasonable?

To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:

There is:

        dev_fmr, dev_fnmr = farfrr(dev_neg, dev_pos, threshold)
        dev_far = dev_fmr * (1 - dev_fta)
        dev_frr = dev_fta + dev_fnmr * (1 - dev_fta)

In my view, FTA is something related to biometrics, so it should computed if --biometrics is passed or on bob.bio.base only.

@andre.anjos pls create a separate issue for this. This issue is more of a meta issue to track other issues. Thanks.

mentioned in issue #46 (closed)

mentioned in issue bob.bio.base#124

mentioned in issue bob.pad.base#26 (closed)

mentioned in issue bob.fusion.base#6

I think we are done with this, right?

closed

Harmonisation of performance reporting

Designs

Child items ...

Activity