bob.measure issueshttps://gitlab.idiap.ch/bob/bob.measure/-/issues2022-11-10T10:14:15Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/69Switch to new CI/CD configuration2022-11-10T10:14:15ZYannick DAYERSwitch to new CI/CD configurationWe need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.Roadmap to the major version of Bob 12https://gitlab.idiap.ch/bob/bob.measure/-/issues/64Partially missing documentation2021-10-29T15:34:56ZLaurent COLBOISPartially missing documentationHi, I noticed the docstrings of many `bob.measure` functions disappeared from the doc between Bob 8 and now, e.g.:
Bob 8
![image](/uploads/30049b39551cd51a5015b20548ee9fd3/image.png)
Current
![image](/uploads/a65659b231b22762783138c1fe...Hi, I noticed the docstrings of many `bob.measure` functions disappeared from the doc between Bob 8 and now, e.g.:
Bob 8
![image](/uploads/30049b39551cd51a5015b20548ee9fd3/image.png)
Current
![image](/uploads/a65659b231b22762783138c1fe541b7e/image.png)
I am pretty suspicious it's linked to the functions that have been wrapped with `@array_jit`, I am guessing the docstring is not transmitted after applying the decorator.
ping @amohammadiAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/63bob measure pure python2021-06-10T11:11:12ZTiago de Freitas Pereirabob measure pure pythonHi @bob,
Following our renovation efforts for Bob, shall we make an effort to port this package to be pure python?
The benefits would be:
- Pure python is more convenient (no platform-dependent) than having a compiled one
- More read...Hi @bob,
Following our renovation efforts for Bob, shall we make an effort to port this package to be pure python?
The benefits would be:
- Pure python is more convenient (no platform-dependent) than having a compiled one
- More readable code. Hence, more people would be willing to contribute
- We would get rid of a blitz dependency (that will die at some point)
The drawbacks:
- We would lose the C++ API (does anyone need that?)
- Extra work
Follow bellow all the functions that would need to be ported.
- bob::measure::farfrr
- bob::measure::precision_recall
- bob::measure::f_score
- bob::measure::correctlyClassifiedPositives
- bob::measure::correctlyClassifiedNegatives
- bob::measure::minimizingThreshold
- bob::measure::eerThreshold
- bob::measure::eerRocch
- bob::measure::minWeightedErrorRateThreshold
- bob::measure::minHterThreshold
- bob::measure::farThreshold
- bob::measure::frrThreshold
- bob::measure::log_values
- bob::measure::meaningfulThresholds(
- bob::measure::roc
- bob::measure::precision_recall_curve
- bob::measure::rocc
- hbob::measure::rocch2eer
- bob::measure::roc_for_far
- bob::measure::ppndf
- bob::measure::det
- bob::measure::epc
Thankshttps://gitlab.idiap.ch/bob/bob.measure/-/issues/61absolute numbers for errors are wrong2020-02-14T16:15:23ZGuillaume HEUSCHabsolute numbers for errors are wrongHi @amohammadi,
As already pointed in the mailing-list (https://groups.google.com/forum/#!topic/bob-devel/RXsX2kgjs1M), the numbers of misclassified and total examples are wrong. Here's an illustration:
```
============== ============...Hi @amohammadi,
As already pointed in the mailing-list (https://groups.google.com/forum/#!topic/bob-devel/RXsX2kgjs1M), the numbers of misclassified and total examples are wrong. Here's an illustration:
```
============== =============== ================
.. Development Evaluation
============== =============== ================
APCER (attack) 7.7% 13.1%
APCER 7.7% 13.1%
BPCER 1.0% 9.7%
ACER 4.3% 11.4%
FTA 0.7% 0.6%
FPR 7.7% (451/1321) 13.1% (749/1571)
FNR 1.0% (13/5880) 9.7% (153/5729)
HTER 4.3% 11.4%
FAR 7.6% 13.0%
FRR 1.7% 10.3%
PRECISION 0.7 0.7
RECALL 1.0 0.9
F1_SCORE 0.8 0.8
```
When you look at FPR on Evaluation set for instance, 749/1571 * 100 = 47.1, which is different from 13.1%. Actually, the total number of examples have been swapped, since 749/5729 * 100 = 13.1 and 153/1571 * 100 = 9.7Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/60ROC plots are not shown correctly sometimes2019-08-19T13:56:42ZAmir MOHAMMADIROC plots are not shown correctly sometimesI have a set of dev and eval scores and currently plotting an roc of them looks like this:
![roc-1](/uploads/4277668b4562f846dd1727b362fde519/roc-1.png)
![roc-2](/uploads/30e83f603e102a8fe46bc0b5d98c5848/roc-2.png)
Here is the command...I have a set of dev and eval scores and currently plotting an roc of them looks like this:
![roc-1](/uploads/4277668b4562f846dd1727b362fde519/roc-1.png)
![roc-2](/uploads/30e83f603e102a8fe46bc0b5d98c5848/roc-2.png)
Here is the command I used: `bin/bob measure roc -vvve scores-{dev,eval} --lines-at 1e-2` and here are the score files: [scores-.zip](/uploads/65459456a61267bb175b7beb555f16cd/scores-.zip)
As you can see the dot in the eval set does not fall on the roc line.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/59MinDCF problem in negative sets with outlier scores2019-02-22T14:14:38ZSaeed SARFJOOMinDCF problem in negative sets with outlier scoresWhen we have an outlier score in negative set, the selected threshold in `bob.measure.min_weighted_error_rate_threshold` function is wrong. For example:
``` python
from bob.measure import min_weighted_error_rate_threshold, farfrr
co...When we have an outlier score in negative set, the selected threshold in `bob.measure.min_weighted_error_rate_threshold` function is wrong. For example:
``` python
from bob.measure import min_weighted_error_rate_threshold, farfrr
cost = 0.99
negatives = [-3, -2, -1, -0.5, 4]
positives = [0.5, 3]
th = min_weighted_error_rate_threshold(negatives, positives, cost, True)
print("threshold: " + str(th))
far, frr = farfrr(negatives, positives, th)
mindcf = (cost*far + (1-cost)*frr)*100
print ("minDCF : " + str(mindcf))
```
In this condition the output will be:
```
threshold: 0.0
minDCF : 19.8
```
minDCF can not be more than 1. In this condition a threshold higher than maximum score must be chosen. e.g., with threshold 5 minDCF will be 1.Saeed SARFJOOSaeed SARFJOOhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/58ROC --no-semilogx plots are broken2019-08-19T13:56:42ZAmir MOHAMMADIROC --no-semilogx plots are brokenIn ROC plots, when --no-semilogx is provided the y axis values change from `1-FRR` to `FRR` and the plot looks very similar to a DET curve which is the expected behavior.
However, the dots shown on the plot (activated using --lines-at 1...In ROC plots, when --no-semilogx is provided the y axis values change from `1-FRR` to `FRR` and the plot looks very similar to a DET curve which is the expected behavior.
However, the dots shown on the plot (activated using --lines-at 1e-3,1e-4,...) are still drawn using the `1-FRR` values. Hence the dots do not fall on the ROC plots.
![roc-1](/uploads/9b4f42423d0d6b5b60abc4034f299411/roc-1.png)
![roc-2](/uploads/47172700f57876cae4185373578d6969/roc-2.png)
I have observed this behavior in `bob bio roc` but since it's implemented here, I suspect the bug is in here.https://gitlab.idiap.ch/bob/bob.measure/-/issues/57The test bob.measure.test_script.test_hist_legends stuck the nightlies2018-10-16T08:23:27ZTiago de Freitas PereiraThe test bob.measure.test_script.test_hist_legends stuck the nightliesVery often the test `bob.measure.test_script.test_hist_legends` get stuck in the nightlies build no matter the platform.
Check:
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150561
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jo...Very often the test `bob.measure.test_script.test_hist_legends` get stuck in the nightlies build no matter the platform.
Check:
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150561
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150497
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150312https://gitlab.idiap.ch/bob/bob.measure/-/issues/56Limitations of matplotlib's constrained layout are not taken into account2018-08-27T14:55:54ZAmir MOHAMMADILimitations of matplotlib's constrained layout are not taken into accountPlease see: https://matplotlib.org/tutorials/intermediate/constrainedlayout_guide.html#limitations
Basically we should not be using `plt.subplot` and should be using `GridSpec`.
I think this was recently added to the documentation so w...Please see: https://matplotlib.org/tutorials/intermediate/constrainedlayout_guide.html#limitations
Basically we should not be using `plt.subplot` and should be using `GridSpec`.
I think this was recently added to the documentation so we didn't know before.
Another thing that we did not know is that `plt.subplots_adjust` and `plt.tight_layout` should not be used when constrained_layout is True.
In matplotlib 2.2.3 now constrained_layout is automatically is set to False if either of those is used. See: https://github.com/matplotlib/matplotlib/pull/11588/files
I think we should rethink our usage to make sure we don't go out of supported bounds of matplotlib's constrained layout. Probably then, we can remove a lot of hacks that we had to do.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/55conda build is not using bob-devel2018-07-06T12:53:09ZTiago de Freitas Pereiraconda build is not using bob-develCan I have the eyes of someone on this, please?
https://gitlab.idiap.ch/bob/bob.measure/-/jobs/142196
There's an issue with `bob.measure` when it tries to link against `openblas`.
Looking at the log you can see that, for some reason, ...Can I have the eyes of someone on this, please?
https://gitlab.idiap.ch/bob/bob.measure/-/jobs/142196
There's an issue with `bob.measure` when it tries to link against `openblas`.
Looking at the log you can see that, for some reason, `bob-devel` (which pins openblas) is not being installed during the conda build and conda-build downloads `blas mkl. Hence, we can't link it.
ThanksAmir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/54Documentation issues2018-07-17T09:11:36ZAndré AnjosDocumentation issuesCurrently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to ...Currently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to FPR.Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/53Logging of score filenames2018-07-09T13:14:44ZAmir MOHAMMADILogging of score filenamesCurrently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swa...Currently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA.F.1-{0,1,2,3,4,5,6,7,8,9}/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64
```
It can become complicated to know which files went where.
The best thing is log these score filenames using the logging module.Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/52Provide an option to change the precision of floating points when they are pr...2019-01-16T14:45:19ZAmir MOHAMMADIProvide an option to change the precision of floating points when they are printedCurrently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command lineCurrently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command lineTheophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/51How to enable a grid on histograms/plots?2019-06-24T14:29:53ZAndré AnjosHow to enable a grid on histograms/plots?My histograms and plots now come out like these:
![idiap-full](/uploads/5aee107437a8999bc06f00ac206bd91d/idiap-full.png)
I wonder if there is an option to activate the "grid" on these, like in this one:
![Screen_Shot_2018-06-27_at_09....My histograms and plots now come out like these:
![idiap-full](/uploads/5aee107437a8999bc06f00ac206bd91d/idiap-full.png)
I wonder if there is an option to activate the "grid" on these, like in this one:
![Screen_Shot_2018-06-27_at_09.15.58](/uploads/b35a253fa7c8058a8c0c680f0e1cfe8b/Screen_Shot_2018-06-27_at_09.15.58.png)https://gitlab.idiap.ch/bob/bob.measure/-/issues/50Missing option to make double-histograms larger or higher2019-06-24T14:40:56ZAndré AnjosMissing option to make double-histograms larger or higherCurrently, when one plots with `--eval`, it results in a situation like this:
![idiap-full](/uploads/dede71ba3bafd45bbdb4e2c3698fb420/idiap-full.png)
While this is nice if you want to squeeze the histograms into a single square block, ...Currently, when one plots with `--eval`, it results in a situation like this:
![idiap-full](/uploads/dede71ba3bafd45bbdb4e2c3698fb420/idiap-full.png)
While this is nice if you want to squeeze the histograms into a single square block, there should an option to "enlarge" the canvas, so that the histograms are not "half-size" in the compression direction. The same goes if the histograms are stacked vertically instead.
Another option that would be cool would be to separate each histogram in a single page, using a multi-page PDF or various output files (if the format is PNG or JPEG). This would allow more plasticity when generating plots for papers.https://gitlab.idiap.ch/bob/bob.measure/-/issues/49Small issues with the new scripts2018-07-09T15:09:55ZAmir MOHAMMADISmall issues with the new scripts* In bob measure det command's help message, --titles help is badly formatted:
```
-ts, --titles TEXT The titles of the plots seperated by commas.
For example, if the figure has two plots,...* In bob measure det command's help message, --titles help is badly formatted:
```
-ts, --titles TEXT The titles of the plots seperated by commas.
For example, if the figure has two plots,
"MyTitleA,MyTitleB" is a possible
inputProvide just a space (-t ) to remove
the titles from figures.
```
where `inputProvide just a space (-t ) to` seems badly formatted.
* As pointed out by @andre.anjos, histograms are better (dev and eval histograms) if you show them under each other.
* (Fixed) ~~in the `fmr_iapmr` plot, values are not in percentage but the labels say %~~Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/48HTER consistency in metrics2018-06-28T07:25:02ZAmir MOHAMMADIHTER consistency in metricsSeveral metrics and plots report HTER but I am not sure it is calculated the same way in all cases.
For examples, `bob measure metrics` and `bob measure epc` both report HTER. However, I don't think they are calculated the same way.
Wh...Several metrics and plots report HTER but I am not sure it is calculated the same way in all cases.
For examples, `bob measure metrics` and `bob measure epc` both report HTER. However, I don't think they are calculated the same way.
What should be HTER anyway @bob? at least in biometrics, we have FMR, FNMR, FAR, and FRR.
Is `HTER = (FMR+FNMR)/2` or `HTER = (FAR+FRR)/2`?
Assuming:
```
FAR = FMR * (1 - FTA)
FRR = FTA + FNMR * (1 - FTA)
```https://gitlab.idiap.ch/bob/bob.measure/-/issues/47Histograms do not show legends of the threshold line anymore2018-06-21T13:18:54ZAmir MOHAMMADIHistograms do not show legends of the threshold line anymoreSee `bin/bob measure hist measure_scores/scores-{dev,eval} -e`See `bin/bob measure hist measure_scores/scores-{dev,eval} -e`https://gitlab.idiap.ch/bob/bob.measure/-/issues/46This package should only report, by default, generic metrics2018-10-16T06:35:54ZAndré AnjosThis package should only report, by default, generic metrics> This issue was opened from the discussion on #38. In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.
To reply correctly, I need to know if there are any practical di...> This issue was opened from the discussion on #38. In order to make bob more useful for other tasks unrelated to biometrics, it would be beneficial to address this issue.
To reply correctly, I need to know if there are any practical differences between FAR and FMR, and FRR and FNMR. If there aren't, I would report, on a given threshold:
* FPR -> `False Positive Rate` (spell it out so there are no confusions)
* FNR -> `False Negative Rate`
* Precision
* Recall
* F1-Score
And that is it
An option could allow you, for example, to replace the values above by something more digestible for biometrics, say `--biometrics` and then the program prints:
* False Acceptance Rate
* False Rejection Rate
* Half-Total Error Rate
The thresholding should also be configurable if that is not already the case: it should be possible to say "report me all values when FPR is set to 10%" or "report me all values when FAR is set to 0.01%" or "report me all values at the Equal-Error Rate":
* `bob measure metrics dev-1.txt` (as per above, use Equal-Error Rate to calculate the threshold, that should be also reported)
* `bob measure metrics --biometrics dev-1.txt` (as per above)
* `bob measure metrics --far=0.0001 --biometrics dev-1.txt` (reports values for an FAR of 0.0001 (0.01%), no minimisation takes place)
* `bob measure metrics --criterion=minhter dev-1.txt` (reports values using FPR/FNR terminology with threshold calculated by minimizing the HTER on the set)
For obvious reasons, options such as `--criterion` and `--far` should be mutually exclusive. As it is currently coded, it is confusing that you should pass `--criterion=far --far-value=0.0001`. It would be easier to say `--far=0.0001` and that is it. If the user passes both, then an error is raised.
It is important that this program is very clear about metrics being used, so I would avoid any acronyms during the error reporting. It is OK to have acronyms on the option names, but documentation should be explicit.
Does that sound reasonable?Theophile GENTILHOMMETheophile GENTILHOMMEhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/45nbins has no effect except for first system2018-06-21T11:56:32ZAmir MOHAMMADInbins has no effect except for first systemSee:
```
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,2
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,200
```
These two commands produce the same output figure.
Sorry for describing the issues very sho...See:
```
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,2
bin/bob measure hist measure_scores/scores-{dev,eval} -b 2,2,2,200
```
These two commands produce the same output figure.
Sorry for describing the issues very shortly.Theophile GENTILHOMMETheophile GENTILHOMME