Skip to content
Snippets Groups Projects
Closed ROC and DET plots are calculated incorrectly sometimes
  • View options
  • ROC and DET plots are calculated incorrectly sometimes

  • View options
  • Closed Issue created

    Following the discussion here: https://groups.google.com/forum/#!topic/bob-devel/EIp1nvw5-vQ

    Looks like we have a corner case where the scores have a very large peak in their distribution: hist_data The scores are also available: fusion_all_200_datatset.npy To load them:

    >>> scores  = numpy.load('fusion_all_200_datatset.npy')
    ... positives = scores[0]
    ... negatives = scores[1]
    # The negatives are mostly 0
    >>> sum(negatives < 9e-16)
    51911
    >>> sum(negatives < 9e-17)
    51525
    >>> sum(negatives < 9e-18)
    51029
    >>> sum(negatives < 9e-20)
    49675
    >>> sum(negatives < 9e-22)
    47543
    >>> sum(negatives < 9e-30)
    27487
    >>> sum(negatives < 9e-60)
    0

    @tiago.pereira may be able to provide a set of smaller scores to debug this.

    When I calculate the EER, I get 2.7%:

    >>> scores  = numpy.load('fusion_all_200_datatset.npy')
    ... positives = scores[0]
    ... negatives = scores[1]
    ... 
    >>> threshold = bob.measure.eer_threshold(negatives, positives)
    ... FAR, FRR = bob.measure.farfrr(negatives, positives, threshold)
    ... 
    >>> FAR, FRR
    (0.02762483029114497, 0.027626084163186636)
    >>> negatives.mean(), negatives.std(), positives.mean(), positives.std()
    (3.0290521114232657e-06, 0.00067550514259906739, 0.20795613959959214, 0.32512541269459205)
    >>> 100*(FAR+FRR)/2
    2.76254572271658
    >>> bob.measure.plot.roc(negatives, positives, npoints)
    [<matplotlib.lines.Line2D object at 0x7f472a8bed10>]
    >>> bob.measure.plot.det(negatives, positives, npoints, color=(0,0,0), linestyle='-', label='test')
    ... bob.measure.plot.det_axis([0.1, 80, 0.1, 80])
    ... 
    [-3.090232246772911, 0.8416212348748217, -3.090232246772911, 0.8416212348748217]
    # plot the EER point on the DET curve
    >>> pyplot.plot(bob.measure.ppndf(FAR), bob.measure.ppndf(FAR), 'ro')
    [<matplotlib.lines.Line2D object at 0x7f472a4dc510>]

    But when I plot the ROC and DET curves, I get curves with EER of 6% or more than 20%:

    DET CURVE with around 6% EER: wrong_det

    ROC CURVE with more than 20% EER: wrong_roc

    Linked items ... 0

  • Activity

    • All activity
    • Comments only
    • History only
    • Newest first
    • Oldest first
    Loading Loading Loading Loading Loading Loading Loading Loading Loading Loading