bob.measure issueshttps://gitlab.idiap.ch/bob/bob.measure/-/issues2018-06-21T08:09:28Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/44The legends come out wrong in hist plots after adding the dev option2018-06-21T08:09:28ZAmir MOHAMMADIThe legends come out wrong in hist plots after adding the dev optionpls see:
```
bin/bob pad eval -e /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA_F/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64 -l swan-fac...pls see:
```
bin/bob pad eval -e /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA_F/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64 -l swan-face-pad.rst
```
which generates:
[eval_plots.pdf](/uploads/e22fdb2fa2067dc2aa996f48f95e5030/eval_plots.pdf)
Maybe we should multiply legends by 2?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/43--log option does not work in the evaluate command2018-06-21T08:07:07ZAmir MOHAMMADI--log option does not work in the evaluate commandSee:
```
bin/bob measure evaluate measure_scores/scores-{dev,eval} -e -l /tmp/log.txt
Traceback (most recent call last):
File "bin/bob", line 29, in <module>
sys.exit(bob.extension.scripts.main_cli())
File "/home/amir/miniconda/e...See:
```
bin/bob measure evaluate measure_scores/scores-{dev,eval} -e -l /tmp/log.txt
Traceback (most recent call last):
File "bin/bob", line 29, in <module>
sys.exit(bob.extension.scripts.main_cli())
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/commands.py", line 258, in evaluate
ctx.invoke(metrics, scores=scores, evaluation=evaluation)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/commands.py", line 41, in metrics
process = figure.Metrics(ctx, scores, evaluation, load.split)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/figure.py", line 170, in __init__
self.log_file = open(self._log, self._open_mode)
TypeError: open() argument 2 must be str, not None
```Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/42The evaluate command prints all the metrics2018-06-20T13:11:00ZAmir MOHAMMADIThe evaluate command prints all the metricsWhen you run `bob measure evaluate`, it prints all the possible metrics for each system.
I think it would be better to print eer only by default and provide the --criterion option.
This is also true for bob.bio.base and bob.pad.baseWhen you run `bob measure evaluate`, it prints all the possible metrics for each system.
I think it would be better to print eer only by default and provide the --criterion option.
This is also true for bob.bio.base and bob.pad.baseAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/41The hist command needs some adjustments2018-06-20T13:11:00ZAmir MOHAMMADIThe hist command needs some adjustments@theophile.gentilhomme while evaluating the hist command, I realized by default only eval scores are plotted and are asked for. The docs say:
```
If you want to
display dev-scores distributions as well, use ``--show-dev`` option.
```
B...@theophile.gentilhomme while evaluating the hist command, I realized by default only eval scores are plotted and are asked for. The docs say:
```
If you want to
display dev-scores distributions as well, use ``--show-dev`` option.
```
But this --show-dev option does not exist. (I think it should be on by default too)
Please also see https://gitlab.idiap.ch/bob/bob.bio.base/issues/112
* There is no way to disable the threshold line
* You can provide -c far but no --far-value.
* --line-styles does nothing in the hist command. Should it be there at all?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/40--no-semilogx is ignored in ROC curves2018-05-17T12:05:44ZAmir MOHAMMADI--no-semilogx is ignored in ROC curvesPassing `--no-semilogx` has no effect on the output. Maybe the best solution is to add this option to `bob.measure.plot.roc_for_far`.Passing `--no-semilogx` has no effect on the output. Maybe the best solution is to add this option to `bob.measure.plot.roc_for_far`.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/39Define the 4 or 5 column score format as the standard format for bob.measure ...2018-06-27T07:25:30ZAmir MOHAMMADIDefine the 4 or 5 column score format as the standard format for bob.measure and document it.Related to #25 and #37, I think it might be a good idea to define the 4 or 5 column score format as the standard format for bob.measure and document it as opposed to moving it to bob.bio.base. This would allow us to have generic evaluati...Related to #25 and #37, I think it might be a good idea to define the 4 or 5 column score format as the standard format for bob.measure and document it as opposed to moving it to bob.bio.base. This would allow us to have generic evaluation scripts implemented in bob.measure for any machine learning experiment. The score format could be as simple as:
```
claimed_id real_id test_label score
claimed_id model_label real_id test_label score
```
* If `claimed_id` and `real_id` columns are equal, then it's a positive score. Otherwise, it's a negative score.
* The text in columns **should not** contain spaces!
* No need to explain what `claimed_id`, `real_id`, and `model_label` are! We can just mention they are related to biometrics.
* As long as anyone comes up with the scores format above (preferably 4 column format), they can use the scripts in bob.measure.https://gitlab.idiap.ch/bob/bob.measure/-/issues/38Harmonisation of performance reporting2018-10-16T06:34:03ZSébastien MARCELHarmonisation of performance reportingthis is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn
currently we are going to get some support from the devel team led...this is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn
currently we are going to get some support from the devel team led by Samuel @samuel.gaist so it will be good to synch with him as well
---
I have the impression that we are still a little bit behind with respect to the harmonisation of performance reporting as discussed before the Bob refactoring last year.
We are still reporting errors rates and plots with FRR/FAR/SFAR and EER and inconsistently FMR/FNMR/IAPRM/ACPER …
We should converge with an harmonisation following current practices that follow more and more ISO.
I know all the elements are in our hands (Tiago for CMCs, Amir for IAPMR and nice scatter plots with decision, …). See some examples attached.
We need a documented package with examples, on how to produce from a set of scores produced by our biometric and PAD experiments, that anyone can use to report results.
More particularly, we need to use
* FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
* TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
* nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
* nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
* ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
* nice bar plot of score distributions for PAD (BonaFide, PA)
* EPSC for biometric recognition and PAD
* scatter plots for bi-modal biometric recognition
* scatter plots for biometric recognition and PAD
Additionally we would need a routine to compute the statistical significance.
a summary of these performance reporting is provided in the attached document (section 4) prepared with our SWAN partners along with references to ISO documents (that can also be found in our biometrics group directory /idiap/group/biometric/standards/ISO-IEC/ eg. ISO-IEC-19795-1 ).
[EPSC_HTER_w-cnn-motion-fusion.pdf](/uploads/a7cf4621ebc2f1a683877d6ae1841f92/EPSC_HTER_w-cnn-motion-fusion.pdf)
[EPSC_IAPMR_w-cnn-motion-fusion.pdf](/uploads/961054f80567b26f97952bf33bb85b58/EPSC_IAPMR_w-cnn-motion-fusion.pdf)
[gmm_score_distribution_fixed.pdf](/uploads/6ef5976e51f7e896e24000c668944b73/gmm_score_distribution_fixed.pdf)
[ISV_gaussians.pdf](/uploads/a26d62fc94c0c2718a3f952a3d2f7720/ISV_gaussians.pdf)
[TR1-v3-20160930.pdf](/uploads/a8f4dc3ad96cd68576c901f85017b350/TR1-v3-20160930.pdf)
also a nice reference is to look at NIST FRVT ( https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt ) eg. http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=915761 and best practices document attached
in interestingly in ROC/DET when they compare systems they draw lines between points with same threshold !
[060405-BestPractices_v2_1.pdf](/uploads/28acaaea9812fa993bd54351acee8cf6/060405-BestPractices_v2_1.pdf)
SébastienGuillaume HEUSCHGuillaume HEUSCHhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/37Plugin-based scripts for metrics and plots2018-07-05T09:01:47ZAmir MOHAMMADIPlugin-based scripts for metrics and plotswe would like to add a script here:
```sh
$ bob measure
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
This script will also help other packages to implement...we would like to add a script here:
```sh
$ bob measure
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
This script will also help other packages to implement their evaluation scripts.
For example, we could have:
```sh
$ bob measure evaluate
# would print and plot according to FAR and FRR
$ bob bio evaluate
# would print and plot according to FMR and FNMR
$ bob pad evaluate
# would print and plot according to APCER and BPCER
```
We could also have commands that are specific to one plot or metric:
```sh
$ bob measure hter
$ bob measure hist
$ bob bio hist
$ bob pad hist
$ bob pad roc
$ bob face_icb2018 # would plot the figures of bob.paper.face_icb2018
```
It's desirable to be able to chain these commands:
```sh
$ bob measure hter hist roc det
$ bob bio hter hist roc
```
**The original text of this issue is below**
I would like to add two scripts here:
```sh
$ bob metrics
$ bob plots
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
Then, we can augment these scripts in `bob.bio.base` and `bob.pad.base`.
For example, we could have:
```sh
$ bob metrics generic
# would print FAR and FRR
$ bob metrics bio
# would print FMR and FNMR
$ bob metrics pad
# would print APCER and BPCER
$ bob plots hist bio
# would plot histogram of genuines and zei
$ bob plots hist pad
# would plot histogram of bona-fide and PA
$ bob plots hist vuln
# would plot histogram of genuines, zei, and PA for vulnerability analysis
$ bob plots det bio
# would plot det curve with FMR and 1-FNMR
$ bob plots face_icb2018
# would plot the figures of bob.paper.face_icb2018
```
Any package could augment these with setuptools entrypoint and click while `bob.measure` could be a source of generic functions that other packages could take advantage of to easily implement these scripts.
What do you think?
I think this
* Would encourage developers to create similar commands and re-use other's commands.
* Would allow everybody to have its own plot script which easily discoverable by others.
I already have an implementation of this but it's in `bob.pad.base`.
I think it would be better if it was in `bob.measure`.
I would like to possibly tackle other issues while doing this:
* Having ISO conforming plots. bob.bio.base#60 bob.pad.base#1
* Simplifying complicated scripts bob.bio.base#65
* Moving bob.bio metrics and plots out of bob.measure #25Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/36Functions in submodule `load` cannot handle identities or labels with spaces ...2019-06-23T17:12:35ZAndré AnjosFunctions in submodule `load` cannot handle identities or labels with spaces on themThe following score file, for example, cannot be correctly loaded:
```text
id1 id1 name with spaces 1.0
id1 id2 another name with spaces 0.0
```
Inside the submodule `load.py` in this package, we use `csv.read` to read the file content...The following score file, for example, cannot be correctly loaded:
```text
id1 id1 name with spaces 1.0
id1 id2 another name with spaces 0.0
```
Inside the submodule `load.py` in this package, we use `csv.read` to read the file contents. Therefore, I was expecting the following change to work properly:
```text
id1 id1 "name with spaces" 1.0
id1 id2 "another name with spaces" 0.0
```
However, that does not work either because of the way `_estimate_score_file_format()` works - i.e., by using another function to figure out the number of columns than `csv.read`.
If you have suggestions on how to correctly handle this, they are welcome!https://gitlab.idiap.ch/bob/bob.measure/-/issues/35cannnot provide negative values to the apply_threshold script2018-01-16T16:56:34ZAmir MOHAMMADIcannnot provide negative values to the apply_threshold scriptHere is what I have tried so far:
```sh
bin/bob_apply_threshold.py "-0.4221387" ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py -...Here is what I have tried so far:
```sh
bin/bob_apply_threshold.py "-0.4221387" ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py '-0.4221387' ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py "'-0.4221387'" ~/nonorm/scores-dev
cannot convert '-0.4221387' into float for threshold
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py -- -0.4221387 ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
```https://gitlab.idiap.ch/bob/bob.measure/-/issues/34Detection & Identification Curve cannot be right on user's guide2018-01-16T16:56:34ZAndré AnjosDetection & Identification Curve cannot be right on user's guideCan somebody please check?
https://www.idiap.ch/software/bob/docs/bob/bob.measure/master/guide.html#detection-identification-curveCan somebody please check?
https://www.idiap.ch/software/bob/docs/bob/bob.measure/master/guide.html#detection-identification-curveManuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.comhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/33[bob.measure] raspberry pi compilation fails2017-09-22T15:58:07ZFlavio TARSETTI[bob.measure] raspberry pi compilation failsHi all,
Could someone explain to me how to properly compile bob.measure ? I am stuck.
I have this issue:
```
(/home/pi/3dfv/env3dfv)pi@raspberrypi:~/3dfv $ pip install bob.measure
Collecting bob.measure
Using cached bob.measure-2.4....Hi all,
Could someone explain to me how to properly compile bob.measure ? I am stuck.
I have this issue:
```
(/home/pi/3dfv/env3dfv)pi@raspberrypi:~/3dfv $ pip install bob.measure
Collecting bob.measure
Using cached bob.measure-2.4.1.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-iiw7OE/bob.measure/setup.py", line 48, in <module>
boost_modules = boost_modules,
File "/home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/blitz/extension.py", line 52, in __init__
BobExtension.__init__(self, *args, **kwargs)
File "/home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/extension/__init__.py", line 301, in __init__
bob_includes, bob_libraries, bob_library_dirs, bob_macros = get_bob_libraries(self.bob_packages)
File "/home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/extension/__init__.py", line 193, in get_bob_libraries
pkg = importlib.import_module(package)
File "/home/pi/3dfv/env3dfv/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/math/__init__.py", line 6, in <module>
bob.extension.load_bob_library('bob.math', __file__)
File "/home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/extension/__init__.py", line 244, in load_bob_library
ctypes.cdll.LoadLibrary(full_libname)
File "/home/pi/3dfv/env3dfv/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/home/pi/3dfv/env3dfv/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /home/pi/3dfv/env3dfv/lib/python2.7/site-packages/bob/math/libbob_math.so: undefined symbol: dggsvd3_
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-iiw7OE/bob.measure/
```
* I have searched for this dggsvd3_ . It appears to be present only in bob.math in file bob/math/cpp/gsvd.cpp
* Following same request in group https://groups.google.com/forum/#!topic/bob-devel/oYZtVnUaqyM where it's stated that openblas and lapack are required. Both are installed.
* libopenblas-base:
Installed: 0.2.19-3+rpi1
Candidate: 0.2.19-3+rpi1
* libopenblas-dev:
Installed: 0.2.19-3+rpi1
Candidate: 0.2.19-3+rpi1
So could you tell me how to proceed, as this is a blocker right now for me. Thanks in advance.
FlavioAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.measure/-/issues/32Who locked this package???????2017-09-14T17:39:17ZTiago de Freitas PereiraWho locked this package???????https://gitlab.idiap.ch/bob/bob.measure/-/issues/31Build Failed #868352017-09-05T15:08:34ZAndré AnjosBuild Failed #86835@mguenther: I've tried this build twice, and it seems your changes are still failing on Python 3.6 + Mac OSX. Could you please check? https://gitlab.idiap.ch/bob/bob.measure/-/jobs/86835
@tiago.pereira: since we'll be in release mode fr...@mguenther: I've tried this build twice, and it seems your changes are still failing on Python 3.6 + Mac OSX. Could you please check? https://gitlab.idiap.ch/bob/bob.measure/-/jobs/86835
@tiago.pereira: since we'll be in release mode from thursday on, could you please check/help Manuel on this one? He may not have access to a Mac for tests.Manuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.comhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/30Match scores not split up into "genuine" and "impostor" categories correctly ...2017-08-23T16:03:15ZVedrana KRIVOKUCAMatch scores not split up into "genuine" and "impostor" categories correctly for 1vsall UTFVP database protocolThe 1vsall UTFVP database protocol is supposed to compare every fingervein sample to every other sample. The following is an excerpt from the score file:
10_1_1 59_4 0059/0059_4_1_120511-152502 0.23330404
As can be seen above, the mod...The 1vsall UTFVP database protocol is supposed to compare every fingervein sample to every other sample. The following is an excerpt from the score file:
10_1_1 59_4 0059/0059_4_1_120511-152502 0.23330404
As can be seen above, the model ID (10_1_1) and probe ID (59_4) are not in the same format. In particular, the model ID contains client ID + session/finger sample ID, while the probe ID only contains the client ID. This is a problem, because the "_split_scores" function in bob.measure.load, whose purpose is to divide the match scores into "genuine" and "impostor" categories, compares model ID to probe ID directly. This means that model ID will never match probe ID, the consequence of which is that the function will place all match scores into the "impostor" category and we will get zero "genuine" scores. Of course, if we then try to do any evaluation/plotting, the result will be wrong.André AnjosAndré Anjoshttps://gitlab.idiap.ch/bob/bob.measure/-/issues/29CMC documentation os not consistent with the reality2017-09-01T02:08:25ZTiago de Freitas PereiraCMC documentation os not consistent with the realityThe documentation of the function `bob.measure.cmc` (http://pythonhosted.org/bob.measure/py_api.html#bob.measure.cmc) says that the output is
"A 2D float array representing the CMC curve, with the Rank in the first column and the numbe...The documentation of the function `bob.measure.cmc` (http://pythonhosted.org/bob.measure/py_api.html#bob.measure.cmc) says that the output is
"A 2D float array representing the CMC curve, with the Rank in the first column and the number of correctly classified clients (in this rank) in the second column."
This is not what is happening.
The function returns a 1D array representing the CMC curve, with the identification rate sorted by the rank in ascending order.https://gitlab.idiap.ch/bob/bob.measure/-/issues/28Updated API in `bob.math` requires update in this package, too2017-06-30T10:08:32ZManuel Günthersiebenkopf@googlemail.comUpdated API in `bob.math` requires update in this package, tooSee bob/bob.math!13See bob/bob.math!13https://gitlab.idiap.ch/bob/bob.measure/-/issues/27FAR and FRR thresholds are computed even when there is no data support2018-02-19T18:37:26ZManuel Günthersiebenkopf@googlemail.comFAR and FRR thresholds are computed even when there is no data supportI have lately come across a situation, where FAR (and FRR) thresholds were computed, although they should not have been.
Imagine the negative score distribution `[0.5, 0.6, 0.7, 0.8, 0.9, 1., 1., 1., 1., 1.]`. A threshold should now be c...I have lately come across a situation, where FAR (and FRR) thresholds were computed, although they should not have been.
Imagine the negative score distribution `[0.5, 0.6, 0.7, 0.8, 0.9, 1., 1., 1., 1., 1.]`. A threshold should now be computed for `FAR=0.1`. Our current implementation of `bob.measure.far_threshold` will return the threshold `1`. However, this threshold does not give us a false acceptance rate of `0.1`, but of `0.5`. In fact, there is no (data-driven) threshold that would provide a false acceptance rate of `0.1`.
A similar issue arises, when the number of data points is not sufficient for a given threshold to be computed.
From only 10 data points, you cannot provide a (data-driven) threshold for `FAR=0.05`, while our current implementation happily provides one.
There are two possible solutions for this issue.
First, we can simply return a threshold that is *just slightly higher* than the largest negative (or slightly lower than the largest positive when computing FRR threshold). This will indeed provide a solution, but this is not justified by data point and might be arbitrarily wrong, i.e., when applied to other test data.
Instead, we should just return `NaN`, since we really cannot compute a justified threshold for the requested FAR or FRR values.May 2017 HackathonAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/26ROC and DET plots are calculated incorrectly sometimes2018-01-16T16:56:34ZAmir MOHAMMADIROC and DET plots are calculated incorrectly sometimesFollowing the discussion here: https://groups.google.com/forum/#!topic/bob-devel/EIp1nvw5-vQ
Looks like we have a corner case where the scores have a very large peak in their distribution:
![hist_data](/uploads/ee0071522224130f9fa7be...Following the discussion here: https://groups.google.com/forum/#!topic/bob-devel/EIp1nvw5-vQ
Looks like we have a corner case where the scores have a very large peak in their distribution:
![hist_data](/uploads/ee0071522224130f9fa7be7b8508f2e4/hist_data.png)
The scores are also available: [fusion_all_200_datatset.npy](/uploads/fa88359dfdb4d66cdbc54f3e4a677149/fusion_all_200_datatset.npy)
To load them:
```python
>>> scores = numpy.load('fusion_all_200_datatset.npy')
... positives = scores[0]
... negatives = scores[1]
# The negatives are mostly 0
>>> sum(negatives < 9e-16)
51911
>>> sum(negatives < 9e-17)
51525
>>> sum(negatives < 9e-18)
51029
>>> sum(negatives < 9e-20)
49675
>>> sum(negatives < 9e-22)
47543
>>> sum(negatives < 9e-30)
27487
>>> sum(negatives < 9e-60)
0
```
@tiago.pereira may be able to provide a set of smaller scores to debug this.
When I calculate the EER, I get `2.7%`:
```python
>>> scores = numpy.load('fusion_all_200_datatset.npy')
... positives = scores[0]
... negatives = scores[1]
...
>>> threshold = bob.measure.eer_threshold(negatives, positives)
... FAR, FRR = bob.measure.farfrr(negatives, positives, threshold)
...
>>> FAR, FRR
(0.02762483029114497, 0.027626084163186636)
>>> negatives.mean(), negatives.std(), positives.mean(), positives.std()
(3.0290521114232657e-06, 0.00067550514259906739, 0.20795613959959214, 0.32512541269459205)
>>> 100*(FAR+FRR)/2
2.76254572271658
>>> bob.measure.plot.roc(negatives, positives, npoints)
[<matplotlib.lines.Line2D object at 0x7f472a8bed10>]
>>> bob.measure.plot.det(negatives, positives, npoints, color=(0,0,0), linestyle='-', label='test')
... bob.measure.plot.det_axis([0.1, 80, 0.1, 80])
...
[-3.090232246772911, 0.8416212348748217, -3.090232246772911, 0.8416212348748217]
# plot the EER point on the DET curve
>>> pyplot.plot(bob.measure.ppndf(FAR), bob.measure.ppndf(FAR), 'ro')
[<matplotlib.lines.Line2D object at 0x7f472a4dc510>]
```
But when I plot the ROC and DET curves, I get curves with EER of `6%` or more than `20%`:
DET CURVE with around `6%` EER:
![wrong_det](/uploads/6c8c93c1c994f0c256ca1b90087a1484/wrong_det.png)
ROC CURVE with more than `20%` EER:
![wrong_roc](/uploads/4730c633a9c7a6ae0ef0d56227e79e41/wrong_roc.png)May 2017 HackathonAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/25Moving biometrics-related functionality to bob.bio.base2018-10-16T06:34:31ZAndré AnjosMoving biometrics-related functionality to bob.bio.baseAs Bob keeps its trajectory to serve more types of systems, it becomes less and less obvious to keep biometrics-related functionality inside this package. I'm proposing we move those into `bob.bio.base`, which is the place they should ha...As Bob keeps its trajectory to serve more types of systems, it becomes less and less obvious to keep biometrics-related functionality inside this package. I'm proposing we move those into `bob.bio.base`, which is the place they should have been in the first place.
A few things that come to mind:
* All score loading/saving functionality
* OpenBR exchange support
* The scripts, which are tunned for Biometrics-style reporting (and can only load biometric score files)
* Not sure about all the identification stuff, maybe generic enough to keep here?
Thanks for your feedback.May 2017 HackathonGuillaume HEUSCHGuillaume HEUSCH