Commit 40332754 authored by Theophile GENTILHOMME's avatar Theophile GENTILHOMME

Modify/Add the guide according to the previous changes

parent 76ad3291
Pipeline #17956 failed with stage
in 22 minutes and 19 seconds
......@@ -99,7 +99,12 @@ defined in the first equation.
The input to create these two vectors is generated by experiments conducted
by the user and normally sits in files that may need some parsing before
these vectors can be extracted.
these vectors can be extracted. While it is not possible to provide a parser
for every individual file that may be generated in different experimental
frameworks, we do provide a parser for a generic two columns format
where the first column contains -1/1 for negative/positive and the second column
contains score values. Please refer to the documentation of
:py:func:`bob.measure.load.split` for more details.
In the remainder of this section we assume you have successfully parsed and
loaded your scores in two 1D float64 vectors and are ready to evaluate the
......@@ -430,6 +435,149 @@ look at the implementations at :py:mod:`bob.measure.plot` to understand how to
use the |project| methods to compute the curves and interlace that in the way
that best suits you.
Full applications
-----------------
Commands under ``bob measure`` can be used to quickly evaluate a set of
scores and generate plots. We present these commands in this section. The commands
take as input generic 2-column data format as specified in the documentation of
:py:func:`bob.measure.load.split`
Metrics
=======
To calculate the threshold using a certain criterion (EER (default) or min.HTER)
on a set, after setting up |project|, just do:
.. code-block:: sh
$ bob measure metrics dev-1.txt
[Min. criterion: EER] Threshold on Development set `dev-1.txt`: -8.025286e-03
╒══════╤═════════════════════════╕
│ │ Development dev-1.txt │
╞══════╪═════════════════════════╡
│ FMR │ 6.263% (31/495) │
├──────┼─────────────────────────┤
│ FNMR │ 6.208% (28/451) │
├──────┼─────────────────────────┤
│ FAR │ 5.924% │
├──────┼─────────────────────────┤
│ FRR │ 11.273% │
├──────┼─────────────────────────┤
│ HTER │ 8.599% │
╘══════╧═════════════════════════╛
The output will present the threshold together with the FMR, FMNR, FAR, FRR and
HTER on the given set, calculated using such a threshold. The relative counts of FAs
and FRs are also displayed between parenthesis.
.. note::
Several scores files can be given at once and the metrics will be computed
for each of them separatly. Development and test files must be given by
pairs and the ``--test`` (or ``-t``) flag must be given (otherwise test
scores are treated as development scores)
To evaluate the performance of a new score file with a given threshold, use
``--thres``:
.. code-block:: sh
$ bob measure metrics --thres 0.006 test-1.txt
[Min. criterion: user provider] Threshold onDevelopment set `test-1.txt`: -6.000000e-03
╒══════╤═════════════════════════╕
│ │ Development test-1.txt │
╞══════╪═════════════════════════╡
│ FMR │ 5.859% (29/495) │
├──────┼─────────────────────────┤
│ FNMR │ 6.208% (28/451) │
├──────┼─────────────────────────┤
│ FAR │ 5.542% │
├──────┼─────────────────────────┤
│ FRR │ 11.273% │
├──────┼─────────────────────────┤
│ HTER │ 8.408% │
╘══════╧═════════════════════════╛
You can simultaneously conduct the threshold computation and its performance
on a test set:
.. code-block:: sh
$ bob measure metrics --test dev-1.txt test-1.txt
[Min. criterion: EER] Threshold on Development set `dev-1.txt`: -8.025286e-03
╒══════╤═════════════════════════╤═══════════════════╕
│ │ Development dev-1.txt │ Test test-1.txt │
╞══════╪═════════════════════════╪═══════════════════╡
│ FMR │ 6.263% (31/495) │ 5.637% (27/479) │
├──────┼─────────────────────────┼───────────────────┤
│ FNMR │ 6.208% (28/451) │ 6.131% (29/473) │
├──────┼─────────────────────────┼───────────────────┤
│ FAR │ 5.924% │ 5.366% │
├──────┼─────────────────────────┼───────────────────┤
│ FRR │ 11.273% │ 10.637% │
├──────┼─────────────────────────┼───────────────────┤
│ HTER │ 8.599% │ 8.001% │
╘══════╧═════════════════════════╧═══════════════════╛
.. note::
Table format can be changed using ``--tablefmt`` option. Note that
``latex``
format is used by default when the outputs are witten in a log file. See
``bob measure metrics --help`` for more details.
Plots
=====
Customizable plotting commands are available in the :py:mod:`bob.measure` module.
They take a list of development and/or test files and generate a single PDF
file containing the plots. Available plots are:
* ``roc`` (receiver operating characteristic)
* ``det`` (detection error trade-off)
* ``epc`` (expected performance curve)
* ``hist`` (histograms of positive and negatives)
Use the ``--help`` option on the above-cited commands to find-out about more
options.
For example, to generate a DET curve from development and test datasets:
.. code-block:: sh
$bob measure det --test --split --output 'my_det.pdf' dev-1.txt test-1.txt
dev-2.txt test-2.txt
where `my_det.pdf` will contain DET plots for the two experiments.
.. note::
By default, ``det`` and ``roc`` plot development and evaluation curves on
different plots. You can force gather everything in the same plot using
``--no-split`` option.
Evaluate
========
A convenient command `evaluate` is provided to generate multiple metrics and
plots for a list of experiments. It generates two `metrics` outputs with EER
and HTER criteria along with `roc`, `det`, `epc`, `hist` plots for each
experiment. For example:
.. code-block:: sh
$bob measure evaluate -t -l 'my_metrics.txt' -o 'my_plots.pdf' {sys1, sys2}/
{test,dev}
will output metrics and plots for the two experiments (dev and test pairs) in
`my_metrics.txt` and `my_plots.pdf`, respectively.
.. include:: links.rst
.. Place youre references here:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment