From 40332754449301f00eb6cb22de1073b5bcd5f0ba Mon Sep 17 00:00:00 2001
From: Theophile GENTILHOMME <tgentilhomme@jurasix08.idiap.ch>
Date: Thu, 29 Mar 2018 15:35:20 +0200
Subject: [PATCH] Modify/Add the guide according to the previous changes
---
doc/guide.rst | 150 +++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 149 insertions(+), 1 deletion(-)
diff --git a/doc/guide.rst b/doc/guide.rst
index e6f694b..5786df2 100644
--- a/doc/guide.rst
+++ b/doc/guide.rst
@@ -99,7 +99,12 @@ defined in the first equation.
The input to create these two vectors is generated by experiments conducted
by the user and normally sits in files that may need some parsing before
- these vectors can be extracted.
+ these vectors can be extracted. While it is not possible to provide a parser
+ for every individual file that may be generated in different experimental
+ frameworks, we do provide a parser for a generic two columns format
+ where the first column contains -1/1 for negative/positive and the second column
+ contains score values. Please refer to the documentation of
+ :py:func:`bob.measure.load.split` for more details.
In the remainder of this section we assume you have successfully parsed and
loaded your scores in two 1D float64 vectors and are ready to evaluate the
@@ -430,6 +435,149 @@ look at the implementations at :py:mod:`bob.measure.plot` to understand how to
use the |project| methods to compute the curves and interlace that in the way
that best suits you.
+
+Full applications
+-----------------
+
+Commands under ``bob measure`` can be used to quickly evaluate a set of
+scores and generate plots. We present these commands in this section. The commands
+take as input generic 2-column data format as specified in the documentation of
+:py:func:`bob.measure.load.split`
+
+Metrics
+=======
+
+To calculate the threshold using a certain criterion (EER (default) or min.HTER)
+on a set, after setting up |project|, just do:
+
+.. code-block:: sh
+
+ $ bob measure metrics dev-1.txt
+ [Min. criterion: EER] Threshold on Development set `dev-1.txt`: -8.025286e-03
+ ╒══════╤═════════════════════════╕
+ │ │ Development dev-1.txt │
+ ╞══════╪═════════════════════════╡
+ │ FMR │ 6.263% (31/495) │
+ ├──────┼─────────────────────────┤
+ │ FNMR │ 6.208% (28/451) │
+ ├──────┼─────────────────────────┤
+ │ FAR │ 5.924% │
+ ├──────┼─────────────────────────┤
+ │ FRR │ 11.273% │
+ ├──────┼─────────────────────────┤
+ │ HTER │ 8.599% │
+ ╘══════╧═════════════════════════╛
+
+The output will present the threshold together with the FMR, FMNR, FAR, FRR and
+HTER on the given set, calculated using such a threshold. The relative counts of FAs
+and FRs are also displayed between parenthesis.
+
+.. note::
+ Several scores files can be given at once and the metrics will be computed
+ for each of them separatly. Development and test files must be given by
+ pairs and the ``--test`` (or ``-t``) flag must be given (otherwise test
+ scores are treated as development scores)
+
+
+To evaluate the performance of a new score file with a given threshold, use
+``--thres``:
+
+.. code-block:: sh
+
+ $ bob measure metrics --thres 0.006 test-1.txt
+ [Min. criterion: user provider] Threshold onDevelopment set `test-1.txt`: -6.000000e-03
+ ╒══════╤═════════════════════════╕
+ │ │ Development test-1.txt │
+ ╞══════╪═════════════════════════╡
+ │ FMR │ 5.859% (29/495) │
+ ├──────┼─────────────────────────┤
+ │ FNMR │ 6.208% (28/451) │
+ ├──────┼─────────────────────────┤
+ │ FAR │ 5.542% │
+ ├──────┼─────────────────────────┤
+ │ FRR │ 11.273% │
+ ├──────┼─────────────────────────┤
+ │ HTER │ 8.408% │
+ ╘══════╧═════════════════════════╛
+
+
+You can simultaneously conduct the threshold computation and its performance
+on a test set:
+
+.. code-block:: sh
+
+ $ bob measure metrics --test dev-1.txt test-1.txt
+ [Min. criterion: EER] Threshold on Development set `dev-1.txt`: -8.025286e-03
+ ╒══════╤═════════════════════════╤═══════════════════╕
+ │ │ Development dev-1.txt │ Test test-1.txt │
+ ╞══════╪═════════════════════════╪═══════════════════╡
+ │ FMR │ 6.263% (31/495) │ 5.637% (27/479) │
+ ├──────┼─────────────────────────┼───────────────────┤
+ │ FNMR │ 6.208% (28/451) │ 6.131% (29/473) │
+ ├──────┼─────────────────────────┼───────────────────┤
+ │ FAR │ 5.924% │ 5.366% │
+ ├──────┼─────────────────────────┼───────────────────┤
+ │ FRR │ 11.273% │ 10.637% │
+ ├──────┼─────────────────────────┼───────────────────┤
+ │ HTER │ 8.599% │ 8.001% │
+ ╘══════╧═════════════════════════╧═══════════════════╛
+
+.. note::
+ Table format can be changed using ``--tablefmt`` option. Note that
+ ``latex``
+ format is used by default when the outputs are witten in a log file. See
+ ``bob measure metrics --help`` for more details.
+
+
+Plots
+=====
+
+Customizable plotting commands are available in the :py:mod:`bob.measure` module.
+They take a list of development and/or test files and generate a single PDF
+file containing the plots. Available plots are:
+
+* ``roc`` (receiver operating characteristic)
+
+* ``det`` (detection error trade-off)
+
+* ``epc`` (expected performance curve)
+
+* ``hist`` (histograms of positive and negatives)
+
+Use the ``--help`` option on the above-cited commands to find-out about more
+options.
+
+For example, to generate a DET curve from development and test datasets:
+
+.. code-block:: sh
+
+ $bob measure det --test --split --output 'my_det.pdf' dev-1.txt test-1.txt
+ dev-2.txt test-2.txt
+
+where `my_det.pdf` will contain DET plots for the two experiments.
+
+.. note::
+ By default, ``det`` and ``roc`` plot development and evaluation curves on
+ different plots. You can force gather everything in the same plot using
+ ``--no-split`` option.
+
+Evaluate
+========
+
+A convenient command `evaluate` is provided to generate multiple metrics and
+plots for a list of experiments. It generates two `metrics` outputs with EER
+and HTER criteria along with `roc`, `det`, `epc`, `hist` plots for each
+experiment. For example:
+
+.. code-block:: sh
+
+ $bob measure evaluate -t -l 'my_metrics.txt' -o 'my_plots.pdf' {sys1, sys2}/
+ {test,dev}
+
+will output metrics and plots for the two experiments (dev and test pairs) in
+`my_metrics.txt` and `my_plots.pdf`, respectively.
+
+
.. include:: links.rst
.. Place youre references here:
--
GitLab