bob.measure issueshttps://gitlab.idiap.ch/bob/bob.measure/-/issues2024-02-15T13:53:03Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/71No API / function to plot a histogram with scores in memory2024-02-15T13:53:03ZYannick DAYERNo API / function to plot a histogram with scores in memoryThere is no way of creating a score histogram (like `bob bio hist` from bob.bio.base) from code without saving the scores to a file.
`det`, `roc` and other plots commands have a function that allows feeding negative and positive scores ...There is no way of creating a score histogram (like `bob bio hist` from bob.bio.base) from code without saving the scores to a file.
`det`, `roc` and other plots commands have a function that allows feeding negative and positive scores as lists, but `hist` does all the plotting in the Figure class.
We should consider moving the plotting part of the `bob.measure.script.figure.Hist` class in an `hist` function in the `bob.measure.plot` module.https://gitlab.idiap.ch/bob/bob.measure/-/issues/70Output of metrics is confusing when not using an eval set2023-10-17T10:34:57ZYannick DAYEROutput of metrics is confusing when not using an eval setWhen evaluating one file (or multiple files without the `-e`/`--eval` option), the `metrics` option assumes this is a development set (it prints `[Min. criterion: EER ] Threshold on Development set <...>`).
This can be confusing if we w...When evaluating one file (or multiple files without the `-e`/`--eval` option), the `metrics` option assumes this is a development set (it prints `[Min. criterion: EER ] Threshold on Development set <...>`).
This can be confusing if we want to score one file on its own which could not be a development set file (e.g. to get the EER metrics of only an eval set).
suggestions:
- Change the line [script/figure.py:330](https://gitlab.idiap.ch/bob/bob.measure/-/blob/v6.1.0/src/bob/measure/script/figure.py?ref_type=tags#L330) and [342](https://gitlab.idiap.ch/bob/bob.measure/-/blob/v6.1.0/src/bob/measure/script/figure.py?ref_type=tags#L342) to something like `Threshold on file '%s': %s`, and keep the notion of `Development set` only when `self._eval == True`.
- Allow changing the set name with a new option.https://gitlab.idiap.ch/bob/bob.measure/-/issues/68bob.measure.plot.det() yields points out of bounds in some cases2022-05-31T09:21:15ZAlex UNNERVIKbob.measure.plot.det() yields points out of bounds in some casesIt seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
...It seems that `bob.measure.plot.det()` yields incorrect plots where the values are outside of the expected [0,1] range in at least certain cases.
It becomes impossible to plot correctly as a result.
Example code to reproduce the issue:
```
import bob.measure
import matplotlib.pyplot as plt
import numpy as np
plt.figure()
l1 = bob.measure.plot.det([1, 2, 3], [2, 2.5, 4], npoints=200) # works for apparently any number of points, including 3 and 2000
print(np.max((l1[0].get_ydata()))) # yields 8.126357928110227
print(np.min((l1[0].get_ydata()))) # yields -8.126357928110227
print(np.max((l1[0].get_xdata()))) # yields 8.126357928110227
print(np.min((l1[0].get_xdata()))) # yields -8.126357928110227
```
These values are inconsistent with expected values in the range of [0,1].
The image shows the result of the plot of the above code, simply adding `plt.show()`.![bob_det_plot](/uploads/9f7eb0264fac5629eeee1d4c4c633c6a/bob_det_plot.png)https://gitlab.idiap.ch/bob/bob.measure/-/issues/67rocch and calibration implementations are missing2022-05-12T13:36:56ZAmir MOHAMMADIrocch and calibration implementations are missingsince we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106since we removed bob.math in https://gitlab.idiap.ch/bob/bob.measure/-/merge_requests/106The Great Deprecationhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/66Issue with ROC curve2021-11-29T11:24:22ZTiago de Freitas PereiraIssue with ROC curveHi @andre.anjos,
You've mentioned in our last Bob meeting that there's an issue with the ROC plot in some special cases.
Do you have any file containing the true values and prediction scores that triggers the issue?
ThanksHi @andre.anjos,
You've mentioned in our last Bob meeting that there's an issue with the ROC plot in some special cases.
Do you have any file containing the true values and prediction scores that triggers the issue?
Thankshttps://gitlab.idiap.ch/bob/bob.measure/-/issues/65Error margins on cross validation protocols2021-08-02T08:10:25ZAmir MOHAMMADIError margins on cross validation protocolsMany experiments come with K-Fold cross validation protocols.
Currently, we only have support for displaying the numerical metrics using the `multi-metrics` command.
However, support for plotting ROC and DET curves with proper error marg...Many experiments come with K-Fold cross validation protocols.
Currently, we only have support for displaying the numerical metrics using the `multi-metrics` command.
However, support for plotting ROC and DET curves with proper error margins are missing.
Scikit-learn has an example for ROC with +-1 std in https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html
but this might not be the best approach.
Moreover, in our multi-metrics, we also use mean and std to show errors.https://gitlab.idiap.ch/bob/bob.measure/-/issues/62Visualization in a plot with several curves2020-08-07T08:20:18ZHatef OTROSHIVisualization in a plot with several curvesHi,
I was trying to plot ROC curves of several models with `bob bio roc`. Since there are many curves in a plot, it is hard to understand each one and distinguish them. Below, you can find a sample of such plot:
![magma](/uploads/dc3090...Hi,
I was trying to plot ROC curves of several models with `bob bio roc`. Since there are many curves in a plot, it is hard to understand each one and distinguish them. Below, you can find a sample of such plot:
![magma](/uploads/dc30907b6bc6a3cbbf6c4a13e2d4f08a/magma.png)
I was wondering how I can achieve a better visualization. If we use `--line-styles` option in `bob bio roc` we will have a result as below which is still not clear:
![magma_line](/uploads/6b9e27f89c35dd54c946a0fd5ba48290/magma_line.png)
I was searching if I could find a better color map and change the color map of `matplotlib`. Inspired by [this plot](https://stackoverflow.com/a/35971096), I changed the color map by replacing [this line](https://gitlab.idiap.ch/bob/bob.measure/-/blob/master/bob/measure/utils.py#L141) in `bob.measure` with `cmap = pyplot.cm.get_cmap(name='gist_ncar')`- only changing the color map. Here is the result:
![gist_ncar](/uploads/fb6e7a5b3a442c1c355e6b92ded0cef8/gist_ncar.png)
And if I use `--line-styles` option as well, it results the following plot while the above plot is still more clear:
![gist_ncar_line](/uploads/988767d0c726939fad088b31e7c0ead4/gist_ncar_line.png)