bob.measure issues
https://gitlab.idiap.ch/bob/bob.measure/-/issues
2024-02-15T13:53:03Z
https://gitlab.idiap.ch/bob/bob.measure/-/issues/71
No API / function to plot a histogram with scores in memory
2024-02-15T13:53:03Z
Yannick DAYER
No API / function to plot a histogram with scores in memory
There is no way of creating a score histogram (like `bob bio hist` from bob.bio.base) from code without saving the scores to a file.
`det`, `roc` and other plots commands have a function that allows feeding negative and positive scores ...
There is no way of creating a score histogram (like `bob bio hist` from bob.bio.base) from code without saving the scores to a file.
`det`, `roc` and other plots commands have a function that allows feeding negative and positive scores as lists, but `hist` does all the plotting in the Figure class.
We should consider moving the plotting part of the `bob.measure.script.figure.Hist` class in an `hist` function in the `bob.measure.plot` module.
https://gitlab.idiap.ch/bob/bob.measure/-/issues/70
Output of metrics is confusing when not using an eval set
2023-10-17T10:34:57Z
Yannick DAYER
Output of metrics is confusing when not using an eval set
When evaluating one file (or multiple files without the `-e`/`--eval` option), the `metrics` option assumes this is a development set (it prints `[Min. criterion: EER ] Threshold on Development set <...>`).
This can be confusing if we w...
When evaluating one file (or multiple files without the `-e`/`--eval` option), the `metrics` option assumes this is a development set (it prints `[Min. criterion: EER ] Threshold on Development set <...>`).
This can be confusing if we want to score one file on its own which could not be a development set file (e.g. to get the EER metrics of only an eval set).
suggestions:
- Change the line [script/figure.py:330](https://gitlab.idiap.ch/bob/bob.measure/-/blob/v6.1.0/src/bob/measure/script/figure.py?ref_type=tags#L330) and [342](https://gitlab.idiap.ch/bob/bob.measure/-/blob/v6.1.0/src/bob/measure/script/figure.py?ref_type=tags#L342) to something like `Threshold on file '%s': %s`, and keep the notion of `Development set` only when `self._eval == True`.
- Allow changing the set name with a new option.
https://gitlab.idiap.ch/bob/bob.measure/-/issues/69
Switch to new CI/CD configuration
2022-11-10T10:14:15Z
Yannick DAYER
Switch to new CI/CD configuration
We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requir...
We need to adapt this package to the new CI/CD and package format using citools:
- [x] Modify `pyproject.toml`:
- [x] Add information from `setup.py`,
- [x] Add version from `version.txt`,
- [x] Add requirements from `requirements.txt` and `conda/meta.yaml`,
- [x] Empty `setup.py`:
- Leave the call to `setup()` for compatibility,
- [x] Remove `version.txt`,
- [x] Remove `requirements.txt`,
- [x] Modify `conda/meta.yaml`,
- [x] Import data from `pyproject.toml` (`name`, `version`, ...),
- [x] Add the `source.path` field with value `..`,
- [x] Add the `build.noarch` field with value `python`,
- [x] Edit the `build.script` to only contain `"{{ PYTHON }} -m pip install {{ SRC_DIR }} -vv"`,
- [x] Remove test and documentation commands and comments,
- [x] Modify `.gitlab-ci.yml` to point to citools' `python.yml`,
- Use the fields format instead of the URL,
- [x] Move files to follow the `src` layout:
- [x] the whole `bob` folder to `src/bob/`,
- [x] all the tests in `tests/`,
- [x] the test data files in `tests/data`,
- [x] Edit the tests to load the data correctly, either with `os.path.join(os.path.basename(__file__), "data/xxx.txt")` or `pkg_resources.resource_filename(__name__, "data/xxx.txt")`,
- [x] Activate the `packages` option in `settings -> general -> visibility` in the Gitlab project,
- [x] Edit the latest doc badges to point to the `sphinx` directory in `doc/[...]/master`:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Edit the coverage badges to point to the doc's coverage directory:
- [x] in README.md,
- [x] in the GitLab project settings,
- [x] Ensure the CI pipeline passes.
You can look at [bob.learn.em](https://gitlab.idiap.ch/bob/bob.learn.em) for an example of a ported package.
Roadmap to the major version of Bob 12
https://gitlab.idiap.ch/bob/bob.measure/-/issues/68
bob.measure.plot.det() yields points out of bounds in some cases
2022-05-31T09:21:15Z
Alex UNNERVIK
bob.measure.plot.det() yields points out of bounds in some cases
It seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
...
It seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
```
import bob.measure
import matplotlib.pyplot as plt
import numpy as np
plt.figure()
l1 = bob.measure.plot.det([1, 2, 3], [2, 2.5, 4], npoints=200) # works for apparently any number of points, including 3 and 2000
print(np.max((l1[0].get_ydata()))) # yields 8.126357928110227
print(np.min((l1[0].get_ydata()))) # yields -8.126357928110227
print(np.max((l1[0].get_xdata()))) # yields 8.126357928110227
print(np.min((l1[0].get_xdata()))) # yields -8.126357928110227
```
These values are inconsistent with expected values in the range of [0,1].
The image shows the result of the plot of the above code, simply adding `plt.show()`.![bob_det_plot](/uploads/9f7eb0264fac5629eeee1d4c4c633c6a/bob_det_plot.png)
https://gitlab.idiap.ch/bob/bob.measure/-/issues/67
rocch and calibration implementations are missing
2022-05-12T13:36:56Z
Amir MOHAMMADI
rocch and calibration implementations are missing
since we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106
since we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106
The Great Deprecation
https://gitlab.idiap.ch/bob/bob.measure/-/issues/66
Issue with ROC curve
2021-11-29T11:24:22Z
Tiago de Freitas Pereira
Issue with ROC curve
Hi @andre.anjos,
You've mentioned in our last Bob meeting that there's an issue with the ROC plot in some special cases.
Do you have any file containing the true values and prediction scores that triggers the issue?
Thanks
Hi @andre.anjos,
You've mentioned in our last Bob meeting that there's an issue with the ROC plot in some special cases.
Do you have any file containing the true values and prediction scores that triggers the issue?
Thanks
https://gitlab.idiap.ch/bob/bob.measure/-/issues/65
Error margins on cross validation protocols
2021-08-02T08:10:25Z
Amir MOHAMMADI
Error margins on cross validation protocols
Many experiments come with K-Fold cross validation protocols.
Currently, we only have support for displaying the numerical metrics using the `multi-metrics` command.
However, support for plotting ROC and DET curves with proper error marg...
Many experiments come with K-Fold cross validation protocols.
Currently, we only have support for displaying the numerical metrics using the `multi-metrics` command.
However, support for plotting ROC and DET curves with proper error margins are missing.
Scikit-learn has an example for ROC with +-1 std in https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html
but this might not be the best approach.
Moreover, in our multi-metrics, we also use mean and std to show errors.
https://gitlab.idiap.ch/bob/bob.measure/-/issues/64
Partially missing documentation
2021-10-29T15:34:56Z
Laurent COLBOIS
Partially missing documentation
Hi, I noticed the docstrings of many `bob.measure` functions disappeared from the doc between Bob 8 and now, e.g.:
Bob 8
![image](/uploads/30049b39551cd51a5015b20548ee9fd3/image.png)
Current
![image](/uploads/a65659b231b22762783138c1fe...
Hi, I noticed the docstrings of many `bob.measure` functions disappeared from the doc between Bob 8 and now, e.g.:
Bob 8
![image](/uploads/30049b39551cd51a5015b20548ee9fd3/image.png)
Current
![image](/uploads/a65659b231b22762783138c1fe541b7e/image.png)
I am pretty suspicious it's linked to the functions that have been wrapped with `@array_jit`, I am guessing the docstring is not transmitted after applying the decorator.
ping @amohammadi
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.measure/-/issues/63
bob measure pure python
2021-06-10T11:11:12Z
Tiago de Freitas Pereira
bob measure pure python
Hi @bob,
Following our renovation efforts for Bob, shall we make an effort to port this package to be pure python?
The benefits would be:
- Pure python is more convenient (no platform-dependent) than having a compiled one
- More read...
Hi @bob,
Following our renovation efforts for Bob, shall we make an effort to port this package to be pure python?
The benefits would be:
- Pure python is more convenient (no platform-dependent) than having a compiled one
- More readable code. Hence, more people would be willing to contribute
- We would get rid of a blitz dependency (that will die at some point)
The drawbacks:
- We would lose the C++ API (does anyone need that?)
- Extra work
Follow bellow all the functions that would need to be ported.
- bob::measure::farfrr
- bob::measure::precision_recall
- bob::measure::f_score
- bob::measure::correctlyClassifiedPositives
- bob::measure::correctlyClassifiedNegatives
- bob::measure::minimizingThreshold
- bob::measure::eerThreshold
- bob::measure::eerRocch
- bob::measure::minWeightedErrorRateThreshold
- bob::measure::minHterThreshold
- bob::measure::farThreshold
- bob::measure::frrThreshold
- bob::measure::log_values
- bob::measure::meaningfulThresholds(
- bob::measure::roc
- bob::measure::precision_recall_curve
- bob::measure::rocc
- hbob::measure::rocch2eer
- bob::measure::roc_for_far
- bob::measure::ppndf
- bob::measure::det
- bob::measure::epc
Thanks
https://gitlab.idiap.ch/bob/bob.measure/-/issues/62
Visualization in a plot with several curves
2020-08-07T08:20:18Z
Hatef OTROSHI
Visualization in a plot with several curves
Hi,
I was trying to plot ROC curves of several models with `bob bio roc`. Since there are many curves in a plot, it is hard to understand each one and distinguish them. Below, you can find a sample of such plot:
![magma](/uploads/dc3090...
Hi,
I was trying to plot ROC curves of several models with `bob bio roc`. Since there are many curves in a plot, it is hard to understand each one and distinguish them. Below, you can find a sample of such plot:
![magma](/uploads/dc30907b6bc6a3cbbf6c4a13e2d4f08a/magma.png)
I was wondering how I can achieve a better visualization. If we use `--line-styles` option in `bob bio roc` we will have a result as below which is still not clear:
![magma_line](/uploads/6b9e27f89c35dd54c946a0fd5ba48290/magma_line.png)
I was searching if I could find a better color map and change the color map of `matplotlib`. Inspired by [this plot](https://stackoverflow.com/a/35971096), I changed the color map by replacing [this line](https://gitlab.idiap.ch/bob/bob.measure/-/blob/master/bob/measure/utils.py#L141) in `bob.measure` with `cmap = pyplot.cm.get_cmap(name='gist_ncar')`- only changing the color map. Here is the result:
![gist_ncar](/uploads/fb6e7a5b3a442c1c355e6b92ded0cef8/gist_ncar.png)
And if I use `--line-styles` option as well, it results the following plot while the above plot is still more clear:
![gist_ncar_line](/uploads/988767d0c726939fad088b31e7c0ead4/gist_ncar_line.png)
https://gitlab.idiap.ch/bob/bob.measure/-/issues/61
absolute numbers for errors are wrong
2020-02-14T16:15:23Z
Guillaume HEUSCH
absolute numbers for errors are wrong
Hi @amohammadi,
As already pointed in the mailing-list (https://groups.google.com/forum/#!topic/bob-devel/RXsX2kgjs1M), the numbers of misclassified and total examples are wrong. Here's an illustration:
```
============== ============...
Hi @amohammadi,
As already pointed in the mailing-list (https://groups.google.com/forum/#!topic/bob-devel/RXsX2kgjs1M), the numbers of misclassified and total examples are wrong. Here's an illustration:
```
============== =============== ================
.. Development Evaluation
============== =============== ================
APCER (attack) 7.7% 13.1%
APCER 7.7% 13.1%
BPCER 1.0% 9.7%
ACER 4.3% 11.4%
FTA 0.7% 0.6%
FPR 7.7% (451/1321) 13.1% (749/1571)
FNR 1.0% (13/5880) 9.7% (153/5729)
HTER 4.3% 11.4%
FAR 7.6% 13.0%
FRR 1.7% 10.3%
PRECISION 0.7 0.7
RECALL 1.0 0.9
F1_SCORE 0.8 0.8
```
When you look at FPR on Evaluation set for instance, 749/1571 * 100 = 47.1, which is different from 13.1%. Actually, the total number of examples have been swapped, since 749/5729 * 100 = 13.1 and 153/1571 * 100 = 9.7
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.measure/-/issues/60
ROC plots are not shown correctly sometimes
2019-08-19T13:56:42Z
Amir MOHAMMADI
ROC plots are not shown correctly sometimes
I have a set of dev and eval scores and currently plotting an roc of them looks like this:
![roc-1](/uploads/4277668b4562f846dd1727b362fde519/roc-1.png)
![roc-2](/uploads/30e83f603e102a8fe46bc0b5d98c5848/roc-2.png)
Here is the command...
I have a set of dev and eval scores and currently plotting an roc of them looks like this:
![roc-1](/uploads/4277668b4562f846dd1727b362fde519/roc-1.png)
![roc-2](/uploads/30e83f603e102a8fe46bc0b5d98c5848/roc-2.png)
Here is the command I used: `bin/bob measure roc -vvve scores-{dev,eval} --lines-at 1e-2` and here are the score files: [scores-.zip](/uploads/65459456a61267bb175b7beb555f16cd/scores-.zip)
As you can see the dot in the eval set does not fall on the roc line.
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.measure/-/issues/59
MinDCF problem in negative sets with outlier scores
2019-02-22T14:14:38Z
Saeed SARFJOO
MinDCF problem in negative sets with outlier scores
When we have an outlier score in negative set, the selected threshold in `bob.measure.min_weighted_error_rate_threshold` function is wrong. For example:
``` python
from bob.measure import min_weighted_error_rate_threshold, farfrr
co...
When we have an outlier score in negative set, the selected threshold in `bob.measure.min_weighted_error_rate_threshold` function is wrong. For example:
``` python
from bob.measure import min_weighted_error_rate_threshold, farfrr
cost = 0.99
negatives = [-3, -2, -1, -0.5, 4]
positives = [0.5, 3]
th = min_weighted_error_rate_threshold(negatives, positives, cost, True)
print("threshold: " + str(th))
far, frr = farfrr(negatives, positives, th)
mindcf = (cost*far + (1-cost)*frr)*100
print ("minDCF : " + str(mindcf))
```
In this condition the output will be:
```
threshold: 0.0
minDCF : 19.8
```
minDCF can not be more than 1. In this condition a threshold higher than maximum score must be chosen. e.g., with threshold 5 minDCF will be 1.
Saeed SARFJOO
Saeed SARFJOO
https://gitlab.idiap.ch/bob/bob.measure/-/issues/58
ROC --no-semilogx plots are broken
2019-08-19T13:56:42Z
Amir MOHAMMADI
ROC --no-semilogx plots are broken
In ROC plots, when --no-semilogx is provided the y axis values change from `1-FRR` to `FRR` and the plot looks very similar to a DET curve which is the expected behavior.
However, the dots shown on the plot (activated using --lines-at 1...
In ROC plots, when --no-semilogx is provided the y axis values change from `1-FRR` to `FRR` and the plot looks very similar to a DET curve which is the expected behavior.
However, the dots shown on the plot (activated using --lines-at 1e-3,1e-4,...) are still drawn using the `1-FRR` values. Hence the dots do not fall on the ROC plots.
![roc-1](/uploads/9b4f42423d0d6b5b60abc4034f299411/roc-1.png)
![roc-2](/uploads/47172700f57876cae4185373578d6969/roc-2.png)
I have observed this behavior in `bob bio roc` but since it's implemented here, I suspect the bug is in here.
https://gitlab.idiap.ch/bob/bob.measure/-/issues/57
The test bob.measure.test_script.test_hist_legends stuck the nightlies
2018-10-16T08:23:27Z
Tiago de Freitas Pereira
The test bob.measure.test_script.test_hist_legends stuck the nightlies
Very often the test `bob.measure.test_script.test_hist_legends` get stuck in the nightlies build no matter the platform.
Check:
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150561
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jo...
Very often the test `bob.measure.test_script.test_hist_legends` get stuck in the nightlies build no matter the platform.
Check:
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150561
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150497
- https://gitlab.idiap.ch/bob/bob.nightlies/-/jobs/150312
https://gitlab.idiap.ch/bob/bob.measure/-/issues/56
Limitations of matplotlib's constrained layout are not taken into account
2018-08-27T14:55:54Z
Amir MOHAMMADI
Limitations of matplotlib's constrained layout are not taken into account
Please see: https://matplotlib.org/tutorials/intermediate/constrainedlayout_guide.html#limitations
Basically we should not be using `plt.subplot` and should be using `GridSpec`.
I think this was recently added to the documentation so w...
Please see: https://matplotlib.org/tutorials/intermediate/constrainedlayout_guide.html#limitations
Basically we should not be using `plt.subplot` and should be using `GridSpec`.
I think this was recently added to the documentation so we didn't know before.
Another thing that we did not know is that `plt.subplots_adjust` and `plt.tight_layout` should not be used when constrained_layout is True.
In matplotlib 2.2.3 now constrained_layout is automatically is set to False if either of those is used. See: https://github.com/matplotlib/matplotlib/pull/11588/files
I think we should rethink our usage to make sure we don't go out of supported bounds of matplotlib's constrained layout. Probably then, we can remove a lot of hacks that we had to do.
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.measure/-/issues/55
conda build is not using bob-devel
2018-07-06T12:53:09Z
Tiago de Freitas Pereira
conda build is not using bob-devel
Can I have the eyes of someone on this, please?
https://gitlab.idiap.ch/bob/bob.measure/-/jobs/142196
There's an issue with `bob.measure` when it tries to link against `openblas`.
Looking at the log you can see that, for some reason, ...
Can I have the eyes of someone on this, please?
https://gitlab.idiap.ch/bob/bob.measure/-/jobs/142196
There's an issue with `bob.measure` when it tries to link against `openblas`.
Looking at the log you can see that, for some reason, `bob-devel` (which pins openblas) is not being installed during the conda build and conda-build downloads `blas mkl. Hence, we can't link it.
Thanks
Amir MOHAMMADI
Amir MOHAMMADI
https://gitlab.idiap.ch/bob/bob.measure/-/issues/54
Documentation issues
2018-07-17T09:11:36Z
André Anjos
Documentation issues
Currently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to ...
Currently, we're using FRR/FAR as information about FNR/FPR, which are the ML de-facto standards.
It would be good to scan the documentation of this package as well as function/class docs and change occurrences of FRR to FNR and FAR to FPR.
Theophile GENTILHOMME
Theophile GENTILHOMME
https://gitlab.idiap.ch/bob/bob.measure/-/issues/53
Logging of score filenames
2018-07-09T13:14:44Z
Amir MOHAMMADI
Logging of score filenames
Currently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swa...
Currently when you use bash expansion, it is not possible to know if the correct score file goes into the correct place.
For example, with a complicated bash expansion like:
```
bin/bob pad multi -e -pn 10 /idiap/user/amohammadi/swan/swan_{frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64}/PA.F.1-{0,1,2,3,4,5,6,7,8,9}/scores/scores-{dev,eval} --legends frame-diff-svm,lbp-svm-160,lbp-svm-64,qm-svm-160,qm-svm-64
```
It can become complicated to know which files went where.
The best thing is log these score filenames using the logging module.
Theophile GENTILHOMME
Theophile GENTILHOMME
https://gitlab.idiap.ch/bob/bob.measure/-/issues/52
Provide an option to change the precision of floating points when they are pr...
2019-01-16T14:45:19Z
Amir MOHAMMADI
Provide an option to change the precision of floating points when they are printed
Currently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command line
Currently floats are rounded to 1 floating point and printed.
It would be best to make this an option on the command line
Theophile GENTILHOMME
Theophile GENTILHOMME