baselines.rst 12.8 KB
Newer Older
1
2
3
4


.. _bob.pad.face.baselines:

5

6
7
8
9
===============================
 Executing Baseline Algorithms
===============================

10
11
This section explains how to execute face presentation attack detection (PAD)
algorithms implemented in ``bob.pad.face``.
12
13
14
15
16


Running Baseline Experiments
----------------------------

17
18
To run the baseline PAD experiments, the ``bob pad vanilla-pad`` command is used.
To see the description of the command, you can type in the console:
19
20
21

.. code-block:: sh

Amir MOHAMMADI's avatar
Amir MOHAMMADI committed
22
   $ bob pad vanilla-pad --help
23

24
This command is explained in more detail in :ref:`bob.pad.base`.
25

26
Usually, it is a good idea to have at least verbose level 2 (i.e., calling
Amir MOHAMMADI's avatar
Amir MOHAMMADI committed
27
28
``bob pad vanilla-pad --verbose --verbose``, or the short version
``bob pad vanilla-pad -vv``).
29
30
31

.. note:: **Running in Parallel**

Amir MOHAMMADI's avatar
Amir MOHAMMADI committed
32
   To run the experiments in parallel, you can define a dask client
33
   (multi-processing) configurations as explained in :ref:`running_in_parallel`.
34

35
36
37
38
39
   In short, to run in the Idiap SGE grid, you can simply add the ``-l``
   (``--dask-client``) command line option, with grid configuration parameters
   (e.g. ``-l sge``).
   To run experiments in parallel on the local machine, simply add the
   ``-l local-parallel`` option.
40
41
42


Database setups and baselines are encoded using
43
``configuration-files``, all stored inside the package structure, in
44
45
46
47
48
49
50
the directory ``bob/pad/face/config``. Documentation for each resource
is available on the section :ref:`bob.pad.face.resources`.

.. warning::

   You **cannot** run experiments just by executing the command line
   instructions described in this guide. You **need first** to procure yourself
51
52
53
54
55
56
   the raw data files that correspond to *each* database used here to correctly
   run experiments with those data. Biometric data is considered private and,
   under EU regulations, cannot be distributed without a consent or license.
   You may consult our :ref:`bob.pad.face.resources.databases` resources
   section for checking currently supported databases and accessing download
   links for the raw data files.
57

58
59
   Once the raw data files have been downloaded, unpack the databases carefully
   and take a note of the root directory where they have been unpacked.
60

61
62
   Use the following commands to specify the correct parameters of your dataset
   (see :ref:`bob.pad.face.resources.databases`):
63

64
   .. code-block:: sh
65

66
67
      $ bob config set bob.db.replaymobile.directory /path/to/replayattack-database/
      $ bob config set bob.db.replaymobile.extension .mov
68

69
70
71
   Notice it is rather important to correctly configure the database as
   described above, otherwise ``bob.pad.base`` will not be able to correctly
   load your data.
72
73
74
75
76
77

   Once this step is done, you can proceed with the instructions below.


------------

78
79
.. _bob.pad.face.baselines.replay_attack:

80
81
82
Baselines on REPLAY-ATTACK database
--------------------------------------

83
84
85
86
87
This section summarizes the results of baseline face PAD experiments on the
REPLAY-ATTACK (`replayattack`_) database.
The description of the database-related settings, which are used to run face PAD
baselines on the Replay-Attack is given here
:ref:`bob.pad.face.resources.databases.replay`. To understand the settings in
88
more detail you can check the corresponding configuration file:
89
``bob/pad/face/config/replay_attack.py``.
90
91
92
93
94


LBP features of facial region + SVM classifier
===================================================

95
96
Detailed description of this PAD pipe-line is given at
:ref:`bob.pad.face.resources.face_pad.lbp_svm_replayattack`.
97

98
99
To run this baseline on the `replayattack`_ database, using the ``grandtest``
protocol, execute the following:
100
101
102

.. code-block:: sh

103
    $ bob pad vanilla-pad replay-attack lbp svm-frames -o <PATH_TO_STORE_THE_RESULTS>
104
105
106

.. tip::

107
    If you are at `idiap`_ you can use the SGE grid to speed-up the calculations.
108
109
    Simply add the ``--dask-client sge`` (or ``-l sge``) argument to the above
    command. For example:
110
111
112

    .. code-block:: sh

113
114
115
        $ bob pad vanilla-pad replay-attack lbp svm-frames \
        --output <PATH_TO_STORE_THE_RESULTS> \
        --dask-client idiap
116
117
118
119
120
121
122
123
124

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/lbp_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
125
    bob pad evaluate \
126
127
    <PATH_TO_STORE_THE_RESULTS>/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/scores-eval \
128
    --legends "LBP features of facial region + SVM classifier + REPLAY-ATTACK database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
129
130
131
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
132
133


134
The error rates for `replayattack`_ database are summarized in the table below:
135
136
137
138
139
140
141

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  15.117  |  15.609  |
+-------------------+----------+----------+

142
The ROC curves for this particular experiment can be downloaded from here:
143
144
145
146
147

:download:`ROC curve <img/ROC_lbp_svm_replay_attack.pdf>`

------------

148

149
150
151
Image Quality Measures as features of facial region + SVM classifier
========================================================================

152
Detailed description of this PAD pipeline is given at :ref:`bob.pad.face.resources.face_pad.qm_svm_replayattack`.
153

154
To run this baseline on the `replayattack`_ database, using the ``grandtest`` protocol, execute the following:
155

156
.. code-block:: sh
157

158
    $ bob pad vanilla-pad replay-attack qm svm-frames \
159
    --sub-directory <PATH_TO_STORE_THE_RESULTS>
160

161
.. tip::
162

163
    Similarly to the tip above you can run this baseline in parallel.
164

165
166
To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/qm_svm.py``
167

168
169
To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:
170

171
.. code-block:: sh
172

Anjith GEORGE's avatar
Anjith GEORGE committed
173
174
175
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
176
    --legends "IQM features of facial region + SVM classifier + REPLAY-ATTACK database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
177
178
179
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
180

181
The EER/HTER errors for `replayattack`_ database are summarized in the table below:
182

183
184
185
+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
186
|   ``grandtest``   |  3.987   |  4.571   |
187
188
189
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:
190

191
192
193
:download:`ROC curve <img/ROC_iqm_svm_replay_attack.pdf>`

------------
194
195


196

197
198
199
200
201
202
.. _bob.pad.face.baselines.replay_mobile:

Baselines on REPLAY-MOBILE database
--------------------------------------

This section summarizes the results of baseline face PAD experiments on the `Replay-Mobile`_ database.
203
The description of the database-related settings, which are used to run face PAD baselines on the Replay-Mobile is given here :ref:`bob.pad.face.resources.databases.replay_mobile`. To understand the settings in more detail you can check the corresponding configuration file : ``bob/pad/face/config/replay_mobile.py``.
204
205


206
207
208
209
210
211
212
213
214
215
LBP features of facial region + SVM classifier
========================================================================

Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.lbp_svm_replayattack`.
Note, that the same PAD pipe-line was used to run experiments on the Replay-Attack database.

To run this baseline on the `Replay-Mobile`_ database, using the ``grandtest`` protocol, execute the following:

.. code-block:: sh

216
217
    $ bob pad vanilla-pad replay-mobile lbp svm_frame \
    --output <PATH_TO_STORE_THE_RESULTS>
218
219
220
221
222
223
224
225
226
227
228
229
230

.. tip::

    Similarly to the tip above you can run this baseline in parallel.

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/lbp_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
231
232
233
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
234
    --legends "LBP features of facial region + SVM classifier + Replay-Mobile database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
235
236
237
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
238

239
The EER/HTER errors for the `Replay-Mobile`_ database are summarized in the table below:
240
241
242
243
244
245
246
247
248
249
250
251
252
253

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  13.814  |  17.174  |
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:

:download:`ROC curve <img/ROC_lbp_svm_replay_mobile.pdf>`

------------


254
255
256
257
258
259
260
261
262
263
Image Quality Measures as features of facial region + SVM classifier
========================================================================

Detailed description of this PAD pipe-line is given at :ref:`bob.pad.face.resources.face_pad.qm_svm_replayattack`.
Note, that the same PAD pipe-line was used to run experiments on the Replay-Attack database.

To run this baseline on the `Replay-Mobile`_ database, using the ``grandtest`` protocol, execute the following:

.. code-block:: sh

264
265
    $ bob pad vanilla-pad replay-mobile qm svm-frames \
    --output <PATH_TO_STORE_THE_RESULTS>
266
267
268
269
270
271
272
273
274
275
276
277
278

.. tip::

    Similarly to the tip above you can run this baseline in parallel.

To understand the settings of this baseline PAD experiment you can check the
corresponding configuration file: ``bob/pad/face/config/qm_svm.py``

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
279
280
281
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
282
    --legends "IQM features of facial region + SVM classifier + Replay-Mobile database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
283
284
285
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
286

287
The EER/HTER errors for the `Replay-Mobile`_ database are summarized in the table below:
288
289
290
291

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
292
|   ``grandtest``   |  1.747   |  4.074   |
293
294
295
296
297
298
299
300
+-------------------+----------+----------+

The ROC curves for the particular experiment can be downloaded from here:

:download:`ROC curve <img/ROC_iqm_svm_replay_mobile.pdf>`

------------

301

302

303

David GEISSBUHLER's avatar
David GEISSBUHLER committed
304
305
306
307
308
309
310
311
312
313
314
315
316
.. _bob.pad.face.baselines.other_db:

Baselines on other databases
--------------------------------------

This section summarizes the results of baseline face PAD experiments on other databases.

------------


MIFS database + LBP features of facial region + SVM classifier
========================================================================

David GEISSBUHLER's avatar
David GEISSBUHLER committed
317
To run this baseline on the MIFS database, using the ``grandtest`` protocol, execute the following:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
318
319
320

.. code-block:: sh

321
322
    $ bob pad vanilla-pad mifs lbp svm-frames \
    --output <PATH_TO_STORE_THE_RESULTS>
David GEISSBUHLER's avatar
David GEISSBUHLER committed
323
324
325
326
327
328

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
329
330
331
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
David GEISSBUHLER's avatar
David GEISSBUHLER committed
332
    --legends "LBP features of facial region + SVM classifier + MIFS database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
333
334
335
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
336

337
The EER/HTER errors for the MIFS database are summarized in the table below:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
338
339
340
341
342
343

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  27.143  |  32.353  |
+-------------------+----------+----------+
344

David GEISSBUHLER's avatar
David GEISSBUHLER committed
345
346
347
348
------------


MIFS database + Image Quality Measures as features of facial region + SVM classifier
David GEISSBUHLER's avatar
David GEISSBUHLER committed
349
========================================================================================
David GEISSBUHLER's avatar
David GEISSBUHLER committed
350

David GEISSBUHLER's avatar
David GEISSBUHLER committed
351
To run this baseline on the MIFS database, using the ``grandtest`` protocol, execute the following:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
352
353
354

.. code-block:: sh

355
356
    $ bob pad vanilla-pad mifs qm svm-frames \
    --output <PATH_TO_STORE_THE_RESULTS>
David GEISSBUHLER's avatar
David GEISSBUHLER committed
357
358
359
360
361
362

To evaluate the results computing EER, HTER and plotting ROC you can use the
following command:

.. code-block:: sh

Anjith GEORGE's avatar
Anjith GEORGE committed
363
364
365
    bob pad evaluate \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-dev  \
    <PATH_TO_STORE_THE_RESULTS>/grandtest/scores/scores-eval \
David GEISSBUHLER's avatar
David GEISSBUHLER committed
366
    --legends "IQM features of facial region + SVM classifier + MIFS database" \
Anjith GEORGE's avatar
Anjith GEORGE committed
367
368
369
    -e \
    --criterion eer \
    -o <PATH_TO_STORE_THE_RESULTS>/ROC.pdf
David GEISSBUHLER's avatar
David GEISSBUHLER committed
370

371
The EER/HTER errors for the MIFS database are summarized in the table below:
David GEISSBUHLER's avatar
David GEISSBUHLER committed
372
373
374
375
376
377
378
379
380
381
382

+-------------------+----------+----------+
|      Protocol     |  EER,\%  |  HTER,\% |
+===================+==========+==========+
|   ``grandtest``   |  34.286  |  28.676  |
+-------------------+----------+----------+

------------


.. include:: links.rst