bob.measure issueshttps://gitlab.idiap.ch/bob/bob.measure/-/issues2018-07-17T09:11:36Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/54Documentation issues2018-07-17T09:11:36ZAndré AnjosDocumentation issuesCurrently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to ...Currently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to FPR.Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/53Logging of score filenames2018-07-09T13:14:44ZAmir MOHAMMADILogging of score filenamesCurrently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swa...Currently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA.F.1-{0,1,2,3,4,5,6,7,8,9}/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64
```
It can become complicated to know which files went where.
The best thing is log these score filenames using the logging module.Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/52Provide an option to change the precision of floating points when they are pr...2019-01-16T14:45:19ZAmir MOHAMMADIProvide an option to change the precision of floating points when they are printedCurrently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command lineCurrently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command lineTheophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/51How to enable a grid on histograms/plots?2019-06-24T14:29:53ZAndré AnjosHow to enable a grid on histograms/plots?My histograms and plots now come out like these:
![idiap-full](/uploads/5aee107437a8999bc06f00ac206bd91d/idiap-full.png)
I wonder if there is an option to activate the "grid" on these, like in this one:
![Screen_Shot_2018-06-27_at_09....My histograms and plots now come out like these:
![idiap-full](/uploads/5aee107437a8999bc06f00ac206bd91d/idiap-full.png)
I wonder if there is an option to activate the "grid" on these, like in this one:
![Screen_Shot_2018-06-27_at_09.15.58](/uploads/b35a253fa7c8058a8c0c680f0e1cfe8b/Screen_Shot_2018-06-27_at_09.15.58.png)https://gitlab.idiap.ch/bob/bob.measure/-/issues/50Missing option to make double-histograms larger or higher2019-06-24T14:40:56ZAndré AnjosMissing option to make double-histograms larger or higherCurrently, when one plots with `--eval`, it results in a situation like this:
![idiap-full](/uploads/dede71ba3bafd45bbdb4e2c3698fb420/idiap-full.png)
While this is nice if you want to squeeze the histograms into a single square block, ...Currently, when one plots with `--eval`, it results in a situation like this:
![idiap-full](/uploads/dede71ba3bafd45bbdb4e2c3698fb420/idiap-full.png)
While this is nice if you want to squeeze the histograms into a single square block, there should an option to "enlarge" the canvas, so that the histograms are not "half-size" in the compression direction. The same goes if the histograms are stacked vertically instead.
Another option that would be cool would be to separate each histogram in a single page, using a multi-page PDF or various output files (if the format is PNG or JPEG). This would allow more plasticity when generating plots for papers.https://gitlab.idiap.ch/bob/bob.measure/-/issues/49Small issues with the new scripts2018-07-09T15:09:55ZAmir MOHAMMADISmall issues with the new scripts* In bob measure det command's help message, --titles help is badly formatted:
```
-ts, --titles TEXT The titles of the plots seperated by commas.
For example, if the figure has two plots,...* In bob measure det command's help message, --titles help is badly formatted:
```
-ts, --titles TEXT The titles of the plots seperated by commas.
For example, if the figure has two plots,
"MyTitleA,MyTitleB" is a possible
inputProvide just a space (-t ) to remove
the titles from figures.
```
where `inputProvide just a space (-t ) to` seems badly formatted.
* As pointed out by @andre.anjos, histograms are better (dev and eval histograms) if you show them under each other.
* (Fixed) ~~in the `fmr_iapmr` plot, values are not in percentage but the labels say %~~Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/48HTER consistency in metrics2018-06-28T07:25:02ZAmir MOHAMMADIHTER consistency in metricsSeveral metrics and plots report HTER but I am not sure it is calculated the same way in all cases.
For examples, `bob measure metrics` and `bob measure epc` both report HTER. However, I don't think they are calculated the same way.
Wh...Several metrics and plots report HTER but I am not sure it is calculated the same way in all cases.
For examples, `bob measure metrics` and `bob measure epc` both report HTER. However, I don't think they are calculated the same way.
What should be HTER anyway @bob? at least in biometrics, we have FMR, FNMR, FAR, and FRR.
Is `HTER = (FMR+FNMR)/2` or `HTER = (FAR+FRR)/2`?
Assuming:
```
FAR = FMR * (1 - FTA)
FRR = FTA + FNMR * (1 - FTA)
```https://gitlab.idiap.ch/bob/bob.measure/-/issues/47Histograms do not show legends of the threshold line anymore2018-06-21T13:18:54ZAmir MOHAMMADIHistograms do not show legends of the threshold line anymoreSee `bin/bob measure hist measure_scores/scores-{dev,eval} -e`See `bin/bob measure hist measure_scores/scores-{dev,eval} -e`https://gitlab.idiap.ch/bob/bob.measure/-/issues/46This package should only report, by default, generic metrics2018-10-16T06:35:54ZAndré AnjosThis package should only report, by default, generic metrics> This issue was opened from the discussion on #38. In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.
To reply correctly, I need to know if there are any practical di...> This issue was opened from the discussion on #38. In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.
To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:
* FPR -> `False Positive Rate` (spell it out so there are no confusions)
* FNR -> `False Negative Rate`
* Precision
* Recall
* F1-Score
And that is it
An option could allow you, for example, to replace the values above by something more digestible for biometrics, say `--biometrics` and then the program prints:
* False Acceptance Rate
* False Rejection Rate
* Half-Total Error Rate
The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":
* `bob measure metrics dev-1.txt` (as per above, use Equal-Error Rate to calculate the threshold, that should be also reported)
* `bob measure metrics --biometrics dev-1.txt` (as per above)
* `bob measure metrics --far=0.0001 --biometrics dev-1.txt` (reports values for an FAR of 0.0001 (0.01%), no minimisation takes place)
* `bob measure metrics --criterion=minhter dev-1.txt` (reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)
For obvious reasons, options such as `--criterion` and `--far` should be mutually exclusive. As it is currently coded, it is confusing that you should pass `--criterion=far --far-value=0.0001`. It would be easier to say `--far=0.0001` and that is it. If the user passes both, then an error is raised.
It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.
Does that sound reasonable?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/45nbins has no effect except for first system2018-06-21T11:56:32ZAmir MOHAMMADInbins has no effect except for first systemSee:
```
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,2
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,200
```
These two commands produce the same output figure.
Sorry for describing the issues very sho...See:
```
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,2
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,200
```
These two commands produce the same output figure.
Sorry for describing the issues very shortly.Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/44The legends come out wrong in hist plots after adding the dev option2018-06-21T08:09:28ZAmir MOHAMMADIThe legends come out wrong in hist plots after adding the dev optionpls see:
```
bin/bob pad eval -e /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA_F/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64 -l swan-fac...pls see:
```
bin/bob pad eval -e /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA_F/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64 -l swan-face-pad.rst
```
which generates:
[eval_plots.pdf](/uploads/e22fdb2fa2067dc2aa996f48f95e5030/eval_plots.pdf)
Maybe we should multiply legends by 2?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/43--log option does not work in the evaluate command2018-06-21T08:07:07ZAmir MOHAMMADI--log option does not work in the evaluate commandSee:
```
bin/bob measure evaluate measure_scores/scores-{dev,eval} -e -l /tmp/log.txt
Traceback (most recent call last):
File "bin/bob", line 29, in <module>
sys.exit(bob.extension.scripts.main_cli())
File "/home/amir/miniconda/e...See:
```
bin/bob measure evaluate measure_scores/scores-{dev,eval} -e -l /tmp/log.txt
Traceback (most recent call last):
File "bin/bob", line 29, in <module>
sys.exit(bob.extension.scripts.main_cli())
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/commands.py", line 258, in evaluate
ctx.invoke(metrics, scores=scores, evaluation=evaluation)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/bob36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/commands.py", line 41, in metrics
process = figure.Metrics(ctx, scores, evaluation, load.split)
File "/home/amir/idiap/git/bobs/bob.pad.base/src/bob.measure/bob/measure/script/figure.py", line 170, in __init__
self.log_file = open(self._log, self._open_mode)
TypeError: open() argument 2 must be str, not None
```Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/42The evaluate command prints all the metrics2018-06-20T13:11:00ZAmir MOHAMMADIThe evaluate command prints all the metricsWhen you run `bob measure evaluate`, it prints all the possible metrics for each system.
I think it would be better to print eer only by default and provide the --criterion option.
This is also true for bob.bio.base and bob.pad.baseWhen you run `bob measure evaluate`, it prints all the possible metrics for each system.
I think it would be better to print eer only by default and provide the --criterion option.
This is also true for bob.bio.base and bob.pad.baseAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/41The hist command needs some adjustments2018-06-20T13:11:00ZAmir MOHAMMADIThe hist command needs some adjustments@theophile.gentilhomme while evaluating the hist command, I realized by default only eval scores are plotted and are asked for. The docs say:
```
If you want to
display dev-scores distributions as well, use ``--show-dev`` option.
```
B...@theophile.gentilhomme while evaluating the hist command, I realized by default only eval scores are plotted and are asked for. The docs say:
```
If you want to
display dev-scores distributions as well, use ``--show-dev`` option.
```
But this --show-dev option does not exist. (I think it should be on by default too)
Please also see https://gitlab.idiap.ch/bob/bob.bio.base/issues/112
* There is no way to disable the threshold line
* You can provide -c far but no --far-value.
* --line-styles does nothing in the hist command. Should it be there at all?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/40--no-semilogx is ignored in ROC curves2018-05-17T12:05:44ZAmir MOHAMMADI--no-semilogx is ignored in ROC curvesPassing `--no-semilogx` has no effect on the output. Maybe the best solution is to add this option to `bob.measure.plot.roc_for_far`.Passing `--no-semilogx` has no effect on the output. Maybe the best solution is to add this option to `bob.measure.plot.roc_for_far`.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/39Define the 4 or 5 column score format as the standard format for bob.measure ...2018-06-27T07:25:30ZAmir MOHAMMADIDefine the 4 or 5 column score format as the standard format for bob.measure and document it.Related to #25 and #37, I think it might be a good idea to define the 4 or 5 column score format as the standard format for bob.measure and document it as opposed to moving it to bob.bio.base. This would allow us to have generic evaluati...Related to #25 and #37, I think it might be a good idea to define the 4 or 5 column score format as the standard format for bob.measure and document it as opposed to moving it to bob.bio.base. This would allow us to have generic evaluation scripts implemented in bob.measure for any machine learning experiment. The score format could be as simple as:
```
claimed_id real_id test_label score
claimed_id model_label real_id test_label score
```
* If `claimed_id` and `real_id` columns are equal, then it's a positive score. Otherwise, it's a negative score.
* The text in columns **should not** contain spaces!
* No need to explain what `claimed_id`, `real_id`, and `model_label` are! We can just mention they are related to biometrics.
* As long as anyone comes up with the scores format above (preferably 4 column format), they can use the scripts in bob.measure.https://gitlab.idiap.ch/bob/bob.measure/-/issues/38Harmonisation of performance reporting2018-10-16T06:34:03ZSébastien MARCELHarmonisation of performance reportingthis is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn
currently we are going to get some support from the devel team led...this is a duplicate of a discussion on the biometric ML in Jan 2017 -- where Guillaume Heusch @heusch agreed to lead this as well as with the help of Hannah @hmuckenhirn
currently we are going to get some support from the devel team led by Samuel @samuel.gaist so it will be good to synch with him as well
---
I have the impression that we are still a little bit behind with respect to the harmonisation of performance reporting as discussed before the Bob refactoring last year.
We are still reporting errors rates and plots with FRR/FAR/SFAR and EER and inconsistently FMR/FNMR/IAPRM/ACPER …
We should converge with an harmonisation following current practices that follow more and more ISO.
I know all the elements are in our hands (Tiago for CMCs, Amir for IAPMR and nice scatter plots with decision, …). See some examples attached.
We need a documented package with examples, on how to produce from a set of scores produced by our biometric and PAD experiments, that anyone can use to report results.
More particularly, we need to use
* FNMR(or GMR=1-FNMR) vs FMR instead of FAR/FRR when we report biometric performance (authentication task) in tables (FNMR @ FMR=0.1% or smaller), DET and ROC (EPC case to be discussed)
* TPIR/rank when we report biometric performance (identification task) in tables (TPIR @ FPIR=0.1%) and CMC
* nice bar plots of score distributions for biometric recognition (Genuine, Zero-effort Impostor)
* nice bar plots of score distributions for biometric recognition and PA (Genuine, Zero-effort Impostor, PA) with IAPMR
* ACPER/BPCER instead of FAR/FRR when we report PAD performance in tables, DET and ROC
* nice bar plot of score distributions for PAD (BonaFide, PA)
* EPSC for biometric recognition and PAD
* scatter plots for bi-modal biometric recognition
* scatter plots for biometric recognition and PAD
Additionally we would need a routine to compute the statistical significance.
a summary of these performance reporting is provided in the attached document (section 4) prepared with our SWAN partners along with references to ISO documents (that can also be found in our biometrics group directory /idiap/group/biometric/standards/ISO-IEC/ eg. ISO-IEC-19795-1 ).
[EPSC_HTER_w-cnn-motion-fusion.pdf](/uploads/a7cf4621ebc2f1a683877d6ae1841f92/EPSC_HTER_w-cnn-motion-fusion.pdf)
[EPSC_IAPMR_w-cnn-motion-fusion.pdf](/uploads/961054f80567b26f97952bf33bb85b58/EPSC_IAPMR_w-cnn-motion-fusion.pdf)
[gmm_score_distribution_fixed.pdf](/uploads/6ef5976e51f7e896e24000c668944b73/gmm_score_distribution_fixed.pdf)
[ISV_gaussians.pdf](/uploads/a26d62fc94c0c2718a3f952a3d2f7720/ISV_gaussians.pdf)
[TR1-v3-20160930.pdf](/uploads/a8f4dc3ad96cd68576c901f85017b350/TR1-v3-20160930.pdf)
also a nice reference is to look at NIST FRVT ( https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt ) eg. http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=915761 and best practices document attached
in interestingly in ROC/DET when they compare systems they draw lines between points with same threshold !
[060405-BestPractices_v2_1.pdf](/uploads/28acaaea9812fa993bd54351acee8cf6/060405-BestPractices_v2_1.pdf)
SébastienGuillaume HEUSCHGuillaume HEUSCHhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/37Plugin-based scripts for metrics and plots2018-07-05T09:01:47ZAmir MOHAMMADIPlugin-based scripts for metrics and plotswe would like to add a script here:
```sh
$ bob measure
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
This script will also help other packages to implement...we would like to add a script here:
```sh
$ bob measure
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
This script will also help other packages to implement their evaluation scripts.
For example, we could have:
```sh
$ bob measure evaluate
# would print and plot according to FAR and FRR
$ bob bio evaluate
# would print and plot according to FMR and FNMR
$ bob pad evaluate
# would print and plot according to APCER and BPCER
```
We could also have commands that are specific to one plot or metric:
```sh
$ bob measure hter
$ bob measure hist
$ bob bio hist
$ bob pad hist
$ bob pad roc
$ bob face_icb2018 # would plot the figures of bob.paper.face_icb2018
```
It's desirable to be able to chain these commands:
```sh
$ bob measure hter hist roc det
$ bob bio hter hist roc
```
**The original text of this issue is below**
I would like to add two scripts here:
```sh
$ bob metrics
$ bob plots
```
These scripts will load their subcommands from setuptools entrypoints; similar to how the `bob ...` loads its subcommands.
Then, we can augment these scripts in `bob.bio.base` and `bob.pad.base`.
For example, we could have:
```sh
$ bob metrics generic
# would print FAR and FRR
$ bob metrics bio
# would print FMR and FNMR
$ bob metrics pad
# would print APCER and BPCER
$ bob plots hist bio
# would plot histogram of genuines and zei
$ bob plots hist pad
# would plot histogram of bona-fide and PA
$ bob plots hist vuln
# would plot histogram of genuines, zei, and PA for vulnerability analysis
$ bob plots det bio
# would plot det curve with FMR and 1-FNMR
$ bob plots face_icb2018
# would plot the figures of bob.paper.face_icb2018
```
Any package could augment these with setuptools entrypoint and click while `bob.measure` could be a source of generic functions that other packages could take advantage of to easily implement these scripts.
What do you think?
I think this
* Would encourage developers to create similar commands and re-use other's commands.
* Would allow everybody to have its own plot script which easily discoverable by others.
I already have an implementation of this but it's in `bob.pad.base`.
I think it would be better if it was in `bob.measure`.
I would like to possibly tackle other issues while doing this:
* Having ISO conforming plots. bob.bio.base#60 bob.pad.base#1
* Simplifying complicated scripts bob.bio.base#65
* Moving bob.bio metrics and plots out of bob.measure #25Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/36Functions in submodule `load` cannot handle identities or labels with spaces ...2019-06-23T17:12:35ZAndré AnjosFunctions in submodule `load` cannot handle identities or labels with spaces on themThe following score file, for example, cannot be correctly loaded:
```text
id1 id1 name with spaces 1.0
id1 id2 another name with spaces 0.0
```
Inside the submodule `load.py` in this package, we use `csv.read` to read the file content...The following score file, for example, cannot be correctly loaded:
```text
id1 id1 name with spaces 1.0
id1 id2 another name with spaces 0.0
```
Inside the submodule `load.py` in this package, we use `csv.read` to read the file contents. Therefore, I was expecting the following change to work properly:
```text
id1 id1 "name with spaces" 1.0
id1 id2 "another name with spaces" 0.0
```
However, that does not work either because of the way `_estimate_score_file_format()` works - i.e., by using another function to figure out the number of columns than `csv.read`.
If you have suggestions on how to correctly handle this, they are welcome!https://gitlab.idiap.ch/bob/bob.measure/-/issues/35cannnot provide negative values to the apply_threshold script2018-01-16T16:56:34ZAmir MOHAMMADIcannnot provide negative values to the apply_threshold scriptHere is what I have tried so far:
```sh
bin/bob_apply_threshold.py "-0.4221387" ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py -...Here is what I have tried so far:
```sh
bin/bob_apply_threshold.py "-0.4221387" ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py '-0.4221387' ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py "'-0.4221387'" ~/nonorm/scores-dev
cannot convert '-0.4221387' into float for threshold
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
bin/bob_apply_threshold.py -- -0.4221387 ~/nonorm/scores-dev
Usage: bob_apply_threshold.py [-v...] [options] <threshold> <scores>
bob_apply_threshold.py --help
bob_apply_threshold.py --version
```