bob.measure issueshttps://gitlab.idiap.ch/bob/bob.measure/-/issues2020-02-14T16:15:23Zhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/2The evaluation metric "Area Under Curve" (AUC) is missing2020-02-14T16:15:23ZAndré AnjosThe evaluation metric "Area Under Curve" (AUC) is missing*Created by: tiagofrepereira2012*
Issue copied from here: https://github.com/idiap/bob/issues/181
###########
One well-known evaluation metric for binary classification problems is the Area Under (ROC) Curve: http://en.wikipedia.o...*Created by: tiagofrepereira2012*
Issue copied from here: https://github.com/idiap/bob/issues/181
###########
One well-known evaluation metric for binary classification problems is the Area Under (ROC) Curve: http://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve
It would be nice to have an implementation of it in bob.measure.Amir MOHAMMADIAmir MOHAMMADIhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/3eer_threshold doesn't behave as expected2016-08-04T09:19:21ZAndré Anjoseer_threshold doesn't behave as expected*Created by: siebenkopf*
Hi there,
I have found out that the ``eer_threshold`` function (as well as the ``min_hter_threshold`` function) does not perform as expected in case of highly unbalanced scores. Particularly, when I try to us...*Created by: siebenkopf*
Hi there,
I have found out that the ``eer_threshold`` function (as well as the ``min_hter_threshold`` function) does not perform as expected in case of highly unbalanced scores. Particularly, when I try to use a special low valued error indicator such as `-10000`, the returned threshold is garbage. Consider the following example:
```
positives = [1.]*100 + [-10000.]
negatives = [-1.]*100 + [-10000.]
threshold = bob.measure.eer_threshold(negatives, positives)
print threshold, bob.measure.farfrr(negatives, positives, threshold)
print 0, bob.measure.farfrr(negatives, positives, 0)
```
The outputs of the two print commands are:
```
-4999.5 (0.9900990099009901, 0.009900990099009901)
0 (0.0, 0.009900990099009901)
```
So, the second output, where I estimated a threshold ``0`` myself, is much more suitable than the first output, which was generated by ``eer_threshold`` (the ``min_hter_threshold`` behaves exactly the same).
I would suggest, we should rethink our implementation of the thresholds. I am pretty sure that there exist smarter ways (I had implemented some during my PhD...)
Cheers
Manuelhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/6`bob.measure.cmc` and `bob.measure.recognition_rate` are not prepared for an ...2016-09-29T16:08:53ZAndré Anjos`bob.measure.cmc` and `bob.measure.recognition_rate` are not prepared for an open-set identification*Created by: tiagofrepereira2012*
Today the functions `bob.measure.cmc` and `bob.measure.recognition_rate` does not support a decision threshold as an argument.
This constrain the scope of these functions only for closed-set identif...*Created by: tiagofrepereira2012*
Today the functions `bob.measure.cmc` and `bob.measure.recognition_rate` does not support a decision threshold as an argument.
This constrain the scope of these functions only for closed-set identification tasks.
I suggest to add an optional keyword argument called `threshold` for that purpose in these functions.
This may impact `bob.measure.cmc_four_column` and `bob.measure.cmc_five_column`
https://gitlab.idiap.ch/bob/bob.measure/-/issues/19load_scores extremely memory hungry2021-06-18T12:58:49ZManuel Günthersiebenkopf@googlemail.comload_scores extremely memory hungryThe new implementation of score loading is memory hungry, as it stores the whole score file in memory. For large score files that have long `client_id`'s and `label`'s, this might easily be too much for a normal desktop machine.
To spli...The new implementation of score loading is memory hungry, as it stores the whole score file in memory. For large score files that have long `client_id`'s and `label`'s, this might easily be too much for a normal desktop machine.
To split the score file into positives and negatives, most of the information (for example, the `label`s) is completely irrelevan.
I remember that I have had this problem with an older version of `bob.measure`, and this is why I have implemented the score reading using a generator function (i.e., `yield`'ing the file line by line) instead of keeping all information of the score file at the same time.
I will provide a better alternative of the 'load_scores' function as a generator function, which does not store the whole score file in memory.Manuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.comhttps://gitlab.idiap.ch/bob/bob.measure/-/issues/21Four and Five column format must be specified by the user, although it could ...2017-04-09T14:34:54ZManuel Günthersiebenkopf@googlemail.comFour and Five column format must be specified by the user, although it could be automatically estimatedSo far, the user has two different functions to load/process score files, depending on the format. On the other hand, @amohammadi has implemented a way of automatically estimating the score file format: https://gitlab.idiap.ch/bob/bob.me...So far, the user has two different functions to load/process score files, depending on the format. On the other hand, @amohammadi has implemented a way of automatically estimating the score file format: https://gitlab.idiap.ch/bob/bob.measure/blob/master/bob/measure/load.py#L315, so that the user does not need to specify the format anymore. I think, we should provide generic functions for similar tasks, too, i.e., having:
* [ ] `bob.measure.load.scores(filename, ncolumns=None)`
* [ ] `bob.measure.load.split(filename, ncolumns=None)`
* [ ] `bob.measure.load.cmc(filename, ncolumns=None)`
These functions should use the same way of handling `ncolumns=None`.Manuel Günthersiebenkopf@googlemail.comManuel Günthersiebenkopf@googlemail.com