vanilla_pad_intro.rst 11.4 KB
Newer Older
1 2 3 4 5 6
.. vim: set fileencoding=utf-8 :
.. author: Yannick Dayer <yannick.dayer@idiap.ch>
.. date: 2020-11-27 15:26:02 +01

.. _bob.pad.base.vanilla_pad_intro:

7 8 9
========================================================================
 Vanilla PAD: Introduction to presentation attack detection in practice
========================================================================
10

11 12 13 14

To easily run experiments in PAD, we offer a generic command called ``bob pad pipelines``.
Such CLI command is an entry point to several pipelines, and this documentation will focus on the one called **vanilla-pad**.

Yannick DAYER's avatar
Yannick DAYER committed
15
The following will introduce how a simple experiment can be run with this tool, from the sample data to a set of metrics and plots, as defined in :ref:`bob.pad.base.pad_intro`.
16

17 18 19 20 21 22

Running a biometric experiment with vanilla-pad
===============================================

A PAD experiment consists of taking a set of biometric `bonafide` and `impostor` samples, feeding them to a pipeline, to finally gather the corresponding set of scores for analysis.

Yannick DAYER's avatar
Yannick DAYER committed
23 24 25 26 27
.. figure:: img/vanilla_pad_pipeline.png
   :figwidth: 75%
   :align: center
   :alt: Data is fed to the pipeline either for training (to fit) or for evaluation (to transform and predict).

Yannick DAYER's avatar
Yannick DAYER committed
28
   The pipeline of Transformer(s) and Classifier can be trained (fit) or used to generate a score for each input sample.
Yannick DAYER's avatar
Yannick DAYER committed
29

Yannick DAYER's avatar
Yannick DAYER committed
30
Similarly to ``vanilla-biometrics``, the ``vanilla-pad`` command needs a pipeline configuration argument to specify which experiment to run and a database argument to indicate what data will be used. These can be given with the ``-p`` (``--pipeline``) and ``-d`` (``--database``) options, respectively::
31

32
$ bob pad vanilla-pad [OPTIONS] -p <pipeline> -d <database>
33

Yannick DAYER's avatar
Yannick DAYER committed
34
The different available options can be listed by giving the ``--help`` flag to the command::
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

$ bob pad vanilla-pad --help


Pipeline
--------

The `pipeline` argument given to vanilla-pad can be either a pipeline `resource name`, or a filename pointing to a configuration file defining the ``pipeline`` variable.

A list of existing `resource names` can be listed with::

$ resources.py -t pipeline


Database
--------

Similarly to `pipeline`, the `database` argument can be in the form of a predefined `resource name`, or a filename pointing to a file defining the ``database`` variable.

The list of database `resource names` can be retrieved with::

$ resources.py -t database


Building your own Vanilla PAD pipeline
======================================

The Vanilla PAD pipeline is the backbone of any experiment in this library. It is composed of:

64
   - Transformers: One or multiple instances in series of :py:class:`sklearn.base.BaseEstimator` and :py:class:`sklearn.base.TransformerMixin`. A Transformer takes a sample as input applies a modification on it and outputs the resulting sample. A transformer can be trained before being used.
65

66
   - A classifier: A class implementing the :py:meth:`fit` and :py:meth:`predict` methods. A Classifier takes a sample as input and returns a score. It is possible to train it beforehand with the :py:meth:`fit` method.
67 68 69 70 71


Transformers
------------

Yannick DAYER's avatar
Yannick DAYER committed
72
A Transformer is a class that implements the fit and transform methods, which allow the application of an operation on a sample of data.
73 74
For more details, see :ref:`bob.bio.base.transformer`.

75 76 77 78 79 80 81 82 83 84
Here is a basic stateless Transformer class:

.. code-block:: python

   from sklearn.base import TransformerMixin, BaseEstimator

   class MyTransformer(TransformerMixin, BaseEstimator):

      def fit(self, X, y):
         return self
85

86 87
      def transform(self, X):
         return modify_sample(X)
88

89 90 91 92

Classifier
----------

93 94 95
A Classifier is the final process of a Vanilla PAD pipeline.
Its goal is to decide if a transformed sample given as input is originating from a genuine sample or if an impostor is trying to be recognized as someone else.
The output is a score for each input sample.
96

97 98 99 100 101 102 103 104 105 106
Here is the minimal structure of a classifier:

.. code-block:: python

   class MyClassifier():
      def __init__(self):
         self.state = 0

      def fit(self, X, y):
         self.state = update_state(self.state, X, y)
107

108 109
      def predict(self, X):
         return do_prediction(self.state, X)
110 111 112
      
      def score(self, X):
         return score(self.state, X)
113 114 115 116 117

.. note::

   The easiest method is to use a scikit-learn classifier, like :py:class:`sklearn.svm.SVC`.
   They are compatible with our pipelines, on the condition to wrap them correctly (see :ref:`below <bob.pad.base.using_sklearn_classifiers>`).
118

119 120 121 122

Running an experiment
=====================

Yannick DAYER's avatar
Yannick DAYER committed
123
Two parts of an experiment have to be executed:
124

125 126 127
- **Fit**: labeled data is fed to the system to train the algorithm to recognize attacks and licit proprieties.
- **Predict**: assessing a series of test samples for authenticity, generating a score for each one.

128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147
These steps are chained together in a pipeline object used by the ``vanilla-pad`` command.
To build such a pipeline, the following configuration file can be created:

.. code-block:: python

   from sklearn.pipeline import Pipeline

   my_transformer = MyTransformer()

   my_classifier = MyClassifier()

   pipeline = Pipeline(
      [
         ("my_transformer", my_transformer),
         ("classifier", my_classifier),
      ]
   )

The pipeline can then be executed with the command::

148
$ bob pad vanilla-pad -d my_database_config.py -p my_pipeline_config.py -f score -o output_dir
149 150

When executed with vanilla-pad, every training sample will pass through the pipeline, executing the ``fit`` methods.
151
Then, every sample of the `dev` set (and/or the `eval` set) will be given to the `transform` method of ``my_transformer`` and the result is passed to the ``decision_function`` method of ``my_classifier``.
152
The output of the classifier (scores) is written to a file.
153

154 155
.. note::

156 157
   By default, vanilla-pad expects the classifier to have a `decision_function` method to call for the prediction step. It can be changed with the '-f' switch to the prediction method of your classifier, in our case the `score` method. 
   The usual `decision_function` of scikit-learn is their `predict_proba` method.
158

159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186

.. _bob.pad.base.using_sklearn_classifiers:

Using scikit-learn classifiers
------------------------------

To use an existing scikit-learn Transformer or Classifier, they need to be wrapped with a `SampleWrapper` (using :py:meth:`bob.pipelines.wrap`) to handle our :py:class:`~bob.pipelines.Sample` objects:

.. code-block:: python

   import bob.pipelines
   from sklearn.pipeline import Pipeline
   from sklearn.svm import SVC

   my_transformer = MyTransformer()


   sklearn_classifier = SVC()
   wrapped_classifier = bob.pipelines.wrap(
      ["sample"], sklearn_classifier, fit_extra_arguments=[("y", "is_bonafide")],
   )

   pipeline = Pipeline(
      [
         ("my_transformer", my_transformer),
         ("classifier", wrapped_classifier),
      ]
   )
187 188


Yannick DAYER's avatar
Yannick DAYER committed
189 190 191 192 193
Scores
------

Executing the vanilla-pad pipeline results in a list of scores, one for each
input sample compared against each registered model.
Yannick DAYER's avatar
Yannick DAYER committed
194
Depending on the chosen ScoreWriter, these scores can be in CSV, 4 columns, or
Yannick DAYER's avatar
Yannick DAYER committed
195
5 columns format, or in a custom user-defined format.
Yannick DAYER's avatar
Yannick DAYER committed
196
By default, the scores are written in the specified output directory (pointed to
Yannick DAYER's avatar
Yannick DAYER committed
197 198 199 200 201 202 203 204 205 206 207 208 209
vanilla-pad with the ``-o`` option), and in the 4 columns format.

The scores represent the performance of a system on that data, but are not
easily interpreted as is so evaluation scripts are available to analyze and show
different aspects of the system performance.

.. figure:: img/vanilla_pad_pipeline_with_eval.png
   :figwidth: 75%
   :align: center
   :alt: The data is fed to the vanilla-pad pipeline, which produces scores files. Scripts allow the evaluation with metrics and plots.

   The vanilla-pad pipeline generates score files that can be used with various scripts to evaluate the system performance by computing metrics or drawing plots.

210

211 212 213
Evaluation
----------

214
Once the scores are generated for each class and group, the evaluation tools can be used to assess the performance of the system, by either drawing plots or computing metrics values at specific operation points.
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262

Generally, the operation thresholds are computed on a specific set (development set or `dev`). Then those threshold values are used to compute the system error rates on a separate set (evaluation set or `eval`).

To retrieve the most common metrics values for a spoofing scenario experiment, run the following command:

.. code-block:: none

   $ bob pad metrics -e scores-{dev,eval} --legends ExpA

   Threshold of 11.639561 selected with the bpcer20 criteria
   ======  ========================  ===================
   ExpA    Development scores-dev    Eval. scores-eval
   ======  ========================  ===================
   APCER   5.0%                      5.0%
   BPCER   100.0%                    100.0%
   ACER    52.5%                     52.5%
   ======  ========================  ===================

   Threshold of 3.969103 selected with the eer criteria
   ======  ========================  ===================
   ExpA    Development scores-dev    Eval. scores-eval
   ======  ========================  ===================
   APCER   100.0%                    100.0%
   BPCER   100.0%                    100.0%
   ACER    100.0%                    100.0%
   ======  ========================  ===================

   Threshold of -0.870550 selected with the min-hter criteria
   ======  ========================  ===================
   ExpA    Development scores-dev    Eval. scores-eval
   ======  ========================  ===================
   APCER   100.0%                    100.0%
   BPCER   19.5%                     19.5%
   ACER    59.7%                     59.7%
   ======  ========================  ===================

.. note::
    When evaluation scores are provided, the ``-e`` option (``--eval``) must be passed.
    See metrics --help for further options.


Plots
-----

Customizable plotting commands are available in the :py:mod:`bob.pad.base` module.
They take a list of development and/or evaluation files and generate a single PDF
file containing the plots.

Yannick DAYER's avatar
Yannick DAYER committed
263
Available plots for a spoofing scenario (command ``bob pad``) are:
264

Yannick DAYER's avatar
Yannick DAYER committed
265
*  ``hist`` (Bonafide and PA histograms along with threshold criterion)
266 267 268 269 270 271 272 273 274 275 276

*  ``epc`` (expected performance curve)

*  ``gen`` (Generate random scores)

*  ``roc`` (receiver operating characteristic)

*  ``det`` (detection error trade-off)

*  ``evaluate`` (Summarize all the above commands in one call)

Yannick DAYER's avatar
Yannick DAYER committed
277 278

Available plots for vulnerability analysis (command ``bob vuln``) are:
279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300

*  ``hist`` (Vulnerability analysis distributions)

*  ``epc`` (expected performance curve)

*  ``gen`` (Generate random scores)

*  ``roc`` (receiver operating characteristic)

*  ``det`` (detection error trade-off)

*  ``epsc`` (expected performance spoofing curve)

*  ``fmr_iapmr``  (Plot FMR vs IAPMR)

*  ``evaluate`` (Summarize all the above commands in one call)


Use the ``--help`` option on the above-cited commands to find-out about more
options.


Yannick DAYER's avatar
Yannick DAYER committed
301
For example, to generate an EPC curve from development and evaluation datasets:
302 303 304 305 306 307 308 309

.. code-block:: sh

    $ bob pad epc -e -o 'my_epc.pdf' scores-{dev,eval}

where `my_epc.pdf` will contain EPC curves for all the experiments.

Vulnerability commands require licit and spoof development and evaluation
Yannick DAYER's avatar
Yannick DAYER committed
310
datasets. For example, to generate EPSC curve:
311 312 313 314 315 316 317

.. code-block:: sh

    $ bob vuln epsc -e .../{licit,spoof}/scores-{dev,eval}


.. note::
Yannick DAYER's avatar
Yannick DAYER committed
318 319 320
    IAPMR curve can be plotted along with EPC and EPSC using the ``--iapmr``
    option. 3D EPSC can be generated using the ``--three-d``. See ``metrics
    --help`` for further options.