baselines.rst 12.9 KB
Newer Older
1
2
3
4


.. _bob.pad.face.baselines:

5

6
7
8
9
===============================
 Executing Baseline Algorithms
===============================

10
11
12
13
14
15
16
17
18
19
20
21
This section explains how to execute face presentation attack detection (PAD) algorithms implemented
in ``bob.pad.face``.


Running Baseline Experiments
----------------------------

To run the baseline PAD experiments, the ``spoof.py`` script located in ``bin`` directory is used.
To see the description of the script you can type in the console:

.. code-block:: sh

22
   $ spoof.py --help
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

This script is explained in more detail in :ref:`bob.pad.base.experiments`.

Usually it is a good idea to have at least verbose level 2 (i.e., calling
``spoof.py --verbose --verbose``, or the short version ``spoof.py
-vv``).

.. note:: **Running in Parallel**

   To run the experiments in parallel, you can define an SGE grid or local host
   (multi-processing) configurations as explained in
   :ref:`running_in_parallel`.

   In short, to run in the Idiap SGE grid, you can simply add the ``--grid``
   command line option, with grid configuration parameters. To run experiments in parallel on
   the local machine, simply add a ``--parallel <N>`` option, where ``<N>``
   specifies the number of parallel jobs you want to execute.


Database setups and baselines are encoded using
:ref:`bob.bio.base.configuration-files`, all stored inside the package root, in
the directory ``bob/pad/face/config``. Documentation for each resource
is available on the section :ref:`bob.pad.face.resources`.

.. warning::

   You **cannot** run experiments just by executing the command line
   instructions described in this guide. You **need first** to procure yourself
   the raw data files that correspond to *each* database used here in order to
   correctly run experiments with those data. Biometric data is considered
   private date and, under EU regulations, cannot be distributed without a
   consent or license. You may consult our
   :ref:`bob.pad.face.resources.databases` resources section for checking
   currently supported databases and accessing download links for the raw data
   files.

   Once the raw data files have been downloaded, particular attention should be
   given to the directory locations of those. Unpack the databases carefully
   and annotate the root directory where they have been unpacked.

   Then, carefully read the *Databases* section of
   :ref:`bob.pad.base.installation` on how to correctly setup the
   ``~/.bob_bio_databases.txt`` file.

   Use the following keywords on the left side of the assignment (see
   :ref:`bob.pad.face.resources.databases`):

   .. code-block:: text

      [YOUR_REPLAY_ATTACK_DIRECTORY] = /complete/path/to/replayattack-database/

   Notice it is rather important to use the strings as described above,
   otherwise ``bob.pad.base`` will not be able to correctly load your images.

   Once this step is done, you can proceed with the instructions below.


------------

82
83
.. _bob.pad.face.baselines.replay_attack:

84
85
86
87
Baselines on REPLAY-ATTACK database
--------------------------------------

This section summarizes the results of baseline face PAD experiments on the REPLAY-ATTACK (`replayattack`_) database.
88
The description of the database-related settings, which are used to run face PAD baselines on the Replay-Attack is given here :ref:`bob.pad.face.resources.databases.replay`. To understand the settings in more details you can check the corresponding configuration file : ``bob/pad/face/config/replay_attack.py``.
89
90
91
92
93
94
95


LBP features of facial region + SVM classifier
===================================================

Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.lbp_svm_replayattack`.

96
To run this baseline on the `replayattack`_ database, using the ``grandtest`` protocol, execute the following:
97
98
99

.. code-block:: sh

100
    $ spoof.py replay-attack lbp-svm \
101
102
103
104
105
106
107
108
109
    --sub-directory <PATH_TO_STORE_THE_RESULTS>

.. tip::

    If you are in `idiap`_ you can use SGE grid to speed-up the calculations.
    Simply add ``--grid idiap`` argument to the above command. For example:

    .. code-block:: sh

110
        $ spoof.py replay-attack lbp-svm \
111
112
113
114
115
116
117
118
119
120
121
        --sub-directory <PATH_TO_STORE_THE_RESULTS> \
        --grid idiap

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/lbp_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
122
123
124
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
125
    --legends "LBP features of facial region + SVM classifier + REPLAY-ATTACK database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
126
127
128
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144


The EER/HTER errors for `replayattack`_ database are summarized in the Table below:

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  15.117  |  15.609  |
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:

:download:`ROC curve <img/ROC_lbp_svm_replay_attack.pdf>`

------------

145

146
147
148
Image Quality Measures as features of facial region + SVM classifier
========================================================================

149
Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.qm_svm_replayattack`.
150

151
To run this baseline on the `replayattack`_ database, using the ``grandtest`` protocol, execute the following:
152

153
.. code-block:: sh
154

155
    $ spoof.py replay-attack qm-svm \
156
    --sub-directory <PATH_TO_STORE_THE_RESULTS>
157

158
.. tip::
159

160
    Similarly to the tip above you can run this baseline in parallel.
161

162
163
To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/qm_svm.py``
164

165
166
To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:
167

168
.. code-block:: sh
169

Anjith GEORGE's avatar
Anjith GEORGE committed
170
171
172
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
173
    --legends "IQM features of facial region + SVM classifier + REPLAY-ATTACK database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
174
175
176
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
177

178
The EER/HTER errors for `replayattack`_ database are summarized in the Table below:
179

180
181
182
+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
183
|   ``grandtest``   |  3.987   |  4.571   |
184
185
186
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:
187

188
189
190
:download:`ROC curve <img/ROC_iqm_svm_replay_attack.pdf>`

------------
191
192


193

194
195
196
197
198
199
.. _bob.pad.face.baselines.replay_mobile:

Baselines on REPLAY-MOBILE database
--------------------------------------

This section summarizes the results of baseline face PAD experiments on the `Replay-Mobile`_ database.
200
The description of the database-related settings, which are used to run face PAD baselines on the Replay-Mobile is given here :ref:`bob.pad.face.resources.databases.replay_mobile`. To understand the settings in more details you can check the corresponding configuration file : ``bob/pad/face/config/replay_mobile.py``.
201
202


203
204
205
206
207
208
209
210
211
212
LBP features of facial region + SVM classifier
========================================================================

Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.lbp_svm_replayattack`.
Note, that the same PAD pipe-line was used to run experiments on the Replay-Attack database.

To run this baseline on the `Replay-Mobile`_ database, using the ``grandtest`` protocol, execute the following:

.. code-block:: sh

213
    $ spoof.py replay-mobile lbp-svm \
214
215
216
217
218
219
220
221
222
223
224
225
226
227
    --sub-directory <PATH_TO_STORE_THE_RESULTS>

.. tip::

    Similarly to the tip above you can run this baseline in parallel.

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/lbp_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
228
229
230
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
231
    --legends "LBP features of facial region + SVM classifier + Replay-Mobile database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
232
233
234
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250

The EER/HTER errors for the `Replay-Mobile`_ database are summarized in the Table below:

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  13.814  |  17.174  |
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:

:download:`ROC curve <img/ROC_lbp_svm_replay_mobile.pdf>`

------------


251
252
253
254
255
256
257
258
259
260
Image Quality Measures as features of facial region + SVM classifier
========================================================================

Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.qm_svm_replayattack`.
Note, that the same PAD pipe-line was used to run experiments on the Replay-Attack database.

To run this baseline on the `Replay-Mobile`_ database, using the ``grandtest`` protocol, execute the following:

.. code-block:: sh

261
    $ spoof.py replay-mobile qm-svm \
262
263
264
265
266
267
268
269
270
271
272
273
274
275
    --sub-directory <PATH_TO_STORE_THE_RESULTS>

.. tip::

    Similarly to the tip above you can run this baseline in parallel.

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/qm_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
276
277
278
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
279
    --legends "IQM features of facial region + SVM classifier + Replay-Mobile database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
280
281
282
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
283
284
285
286
287
288

The EER/HTER errors for the `Replay-Mobile`_ database are summarized in the Table below:

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
289
|   ``grandtest``   |  1.747   |  4.074   |
290
291
292
293
294
295
296
297
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:

:download:`ROC curve <img/ROC_iqm_svm_replay_mobile.pdf>`

------------

298

299

300

David GEISSBUHLER's avatar
David GEISSBUHLER committed
301
302
303
304
305
306
307
308
309
310
311
312
313
.. _bob.pad.face.baselines.other_db:

Baselines on other databases
--------------------------------------

This section summarizes the results of baseline face PAD experiments on other databases.

------------


MIFS database + LBP features of facial region + SVM classifier
========================================================================

David GEISSBUHLER's avatar
David GEISSBUHLER committed
314
To run this baseline on the MIFS database, using the ``grandtest`` protocol, execute the following:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
315
316
317
318
319
320
321
322
323
324
325

.. code-block:: sh

    $ spoof.py mifs lbp-svm \
    --sub-directory <PATH_TO_STORE_THE_RESULTS>

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
326
327
328
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
David GEISSBUHLER's avatar
David GEISSBUHLER committed
329
    --legends "LBP features of facial region + SVM classifier + MIFS database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
330
331
332
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
333

David GEISSBUHLER's avatar
David GEISSBUHLER committed
334
The EER/HTER errors for the MIFS database are summarized in the Table below:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
335
336
337
338
339
340

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  27.143  |  32.353  |
+-------------------+----------+----------+
341

David GEISSBUHLER's avatar
David GEISSBUHLER committed
342
343
344
345
------------


MIFS database + Image Quality Measures as features of facial region + SVM classifier
David GEISSBUHLER's avatar
David GEISSBUHLER committed
346
========================================================================================
David GEISSBUHLER's avatar
David GEISSBUHLER committed
347

David GEISSBUHLER's avatar
David GEISSBUHLER committed
348
To run this baseline on the MIFS database, using the ``grandtest`` protocol, execute the following:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
349
350
351
352
353
354
355
356
357
358
359

.. code-block:: sh

    $ spoof.py mifs qm-svm \
    --sub-directory <PATH_TO_STORE_THE_RESULTS>

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
360
361
362
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
David GEISSBUHLER's avatar
David GEISSBUHLER committed
363
    --legends "IQM features of facial region + SVM classifier + MIFS database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
364
365
366
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
David GEISSBUHLER's avatar
David GEISSBUHLER committed
367

David GEISSBUHLER's avatar
David GEISSBUHLER committed
368
The EER/HTER errors for the MIFS database are summarized in the Table below:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
369
370
371
372
373
374
375
376
377
378
379

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  34.286  |  28.676  |
+-------------------+----------+----------+

------------


.. include:: links.rst