Commit 4422a1ac authored by André Anjos's avatar André Anjos 💬

Improve documentation

parent 4011fbe8
This diff is collapsed.
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Manuel Guenther <Manuel.Guenther@idiap.ch>
# Thu May 16 11:41:49 CEST 2013
#
# Copyright (C) 2011-2013 Idiap Research Institute, Martigny, Switzerland
"""Measures for calibration"""
import math
import numpy
def cllr(negatives, positives):
"""cllr(negatives, positives) -> cllr
"""Cost of log likelihood ratio as defined by the Bosaris toolkit
Computes the 'cost of log likelihood ratio' (:math:`C_{llr}`) measure as
given in the Bosaris toolkit
Parameters:
negatives (array): 1D float array that contains the scores of the
"negative" (noise, non-class) samples of your classifier.
Computes the 'cost of log likelihood ratio' (:math:`C_{llr}`) measure as given in the Bosaris toolkit
positives (array): 1D float array that contains the scores of the
"positive" (signal, class) samples of your classifier.
**Parameters:**
``negatives, positives`` : array_like(1D, float)
The scores computed by comparing elements from different classes and the same class, respectively.
Returns:
**Returns**
float: The computed :math:`C_{llr}` value.
``cllr`` : float
The computed :math:`C_{llr}` value.
"""
sum_pos, sum_neg = 0., 0.
for pos in positives:
......@@ -34,19 +38,25 @@ def cllr(negatives, positives):
def min_cllr(negatives, positives):
"""min_cllr(negatives, positives) -> min_cllr
"""Minimum cost of log likelihood ratio as defined by the Bosaris toolkit
Computes the 'minimum cost of log likelihood ratio' (:math:`C_{llr}^{min}`)
measure as given in the bosaris toolkit
Parameters:
negatives (array): 1D float array that contains the scores of the
"negative" (noise, non-class) samples of your classifier.
Computes the 'minimum cost of log likelihood ratio' (:math:`C_{llr}^{min}`) measure as given in the bosaris toolkit
positives (array): 1D float array that contains the scores of the
"positive" (signal, class) samples of your classifier.
**Parameters:**
``negatives, positives`` : array_like(1D, float)
The scores computed by comparing elements from different classes and the same class, respectively.
Returns:
**Returns**
float: The computed :math:`C_{llr}^{min}` value.
``min_cllr`` : float
The computed :math:`C_{llr}^{min}` value.
"""
from bob.math import pavx
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Andre Anjos <andre.anjos@idiap.ch>
# Wed May 25 13:27:46 2011 +0200
#
# Copyright (C) 2011-2013 Idiap Research Institute, Martigny, Switzerland
"""This script runs error analysis on the development and test set scores, in a
four column format:
......
File mode changed from 100755 to 100644
......@@ -25,11 +25,13 @@ extensions = [
'sphinx.ext.intersphinx',
'sphinx.ext.napoleon',
'sphinx.ext.viewcode',
'matplotlib.sphinxext.plot_directive',
]
import sphinx
if sphinx.__version__ >= "1.4.1":
extensions.append('sphinx.ext.imgmath')
imgmath_image_format = 'svg'
else:
extensions.append('sphinx.ext.pngmath')
......
......@@ -45,6 +45,7 @@ formula:
HTER(\tau, \mathcal{D}) = \frac{FAR(\tau, \mathcal{D}) + FRR(\tau, \mathcal{D})}{2} \quad \textrm{[\%]}
where :math:`\mathcal{D}` denotes the dataset used. Since both the FAR and the
FRR depends on the threshold :math:`\tau`, they are strongly related to each
other: increasing the FAR will reduce the FRR and vice-versa. For this reason,
......@@ -70,8 +71,9 @@ optimal threshold :math:`\tau^*` is then computed using different values of
.. math::
\tau^{*} = \arg\!\min_{\tau} \quad \beta \cdot \textrm{FAR}(\tau, \mathcal{D}_{d}) + (1-\beta) \cdot \textrm{FRR}(\tau, \mathcal{D}_{d})
where :math:`\mathcal{D}_{d}` denotes the development set and should be
completely separate to the evaluation set `\mathcal{D}`.
completely separate to the evaluation set :math:`\mathcal{D}`.
Performance for different values of :math:`\beta` is then computed on the test
set :math:`\mathcal{D}_{t}` using the previously derived threshold. Note that
......@@ -373,6 +375,7 @@ The CMC can be calculated from a relatively complex data structure, which define
negatives = numpy.random.normal(0, 1, 19)
cmc_scores.append((negatives, positives))
bob.measure.plot.cmc(cmc_scores, logx=False)
pyplot.grid(True)
pyplot.title('CMC')
pyplot.xlabel('Rank')
pyplot.xticks([1,5,10,20])
......@@ -383,15 +386,23 @@ The CMC can be calculated from a relatively complex data structure, which define
Usually, there is only a single positive score per probe, but this is not a fixed restriction.
.. note::
The complex data structure can be read from our default 4 or 5 column score files using the :py:func:`bob.measure.load.cmc_four_column` or :py:func:`bob.measure.load.cmc_five_column` function.
The complex data structure can be read from our default 4 or 5 column score
files using the :py:func:`bob.measure.load.cmc_four_column` or
:py:func:`bob.measure.load.cmc_five_column` function.
Detection & Identification Curve
================================
The detection & identification curve is designed to evaluate open set identification tasks.
It can be plotted using the :py:func:`bob.measure.plot.detection_identification_curve` function, but it requires at least one open-set probe, i.e., where no corresponding positive score exists, for which the FAR values are computed.
Here, we plot the detection and identification curve for rank 1, so that the recognition rate for FAR=1 will be identical to the rank one :py:func:`bob.measure.recognition_rate` obtained in the CMC plot above.
The detection & identification curve is designed to evaluate open set
identification tasks. It can be plotted using the
:py:func:`bob.measure.plot.detection_identification_curve` function, but it
requires at least one open-set probe, i.e., where no corresponding positive
score exists, for which the FAR values are computed. Here, we plot the
detection and identification curve for rank 1, so that the recognition rate for
FAR=1 will be identical to the rank one :py:func:`bob.measure.recognition_rate`
obtained in the CMC plot above.
.. plot::
......@@ -423,9 +434,10 @@ The methods inside :py:mod:`bob.measure.plot` are only provided as a
`Matplotlib`_ wrapper to equivalent methods in :py:mod:`bob.measure` that can
only calculate the points without doing any plotting. You may prefer to tweak
the plotting or even use a different plotting system such as gnuplot. Have a
look at the implementations at :py:mod:`bob.measure.plot` to understand how
to use the |project| methods to compute the curves and interlace that in the
way that best suits you.
look at the implementations at :py:mod:`bob.measure.plot` to understand how to
use the |project| methods to compute the curves and interlace that in the way
that best suits you.
Full applications
-----------------
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment