This section explains how to run a complete face PAD experiment using multi-channel autoencoder-based face PAD system, as well as a training work-flow.
The system discussed in this section is introduced the following publication [NGM19]_. It is **strongly recommended** to check the publication for better understanding
of the described work-flow.
.. warning::
Algorithms introduced in this section might be in the process of publishing. Therefore, it is not
allowed to publish results introduced in this section without permission of the owner of the package.
If you are planning to use the results from this section, please contact the owner of the package first.
Please check the ``setup.py`` for contact information.
Running face PAD Experiments
------------------------------
Please refer to :ref:`bob.pad.face.baselines` section of current documentation for more details on how to run the face PAD experiments and setup the databases.
Training multi-channel autoencoder-based face PAD system.
In the above table, **BF** and **PA** stands for samples from **bona-fide** and **presentation attack** classes.
As one can conclude from the table, CelebA and WMCA databases must be installed before the training can take place.
See :ref:`bob.pad.face.baselines` for databases installation details.
1. Train N AEs on RGB data from CelebA
===========================================
In [NGM19]_ N autoencoders are trained, one for each facial region, here for explanatory purposes, a system containing **one** autoencoder is observed, thus N=1.
This autoencoder is first pre-trained using RGB images of entire face, which are cropped from CelebA database.
To prepare the training data one can use the following command:
.. code-block:: sh
./bin/spoof.py \ # spoof.py is used to run the preprocessor
--grid idiap \ # use grid, only for Idiap users, remove otherwise
--groups train \ # preprocess only training set of CelebA
--preprocessor rgb-face-detect-check-quality-128x128 \ # preprocessor entry point
--sub-directory <PATH_TO_STORE_THE_RESULTS> # define your path here
Running above command, the RGB facial images are aligned and cropped from the training set of the CelebA database. Additionally, a quality assessment is applied to each facial image.
More specifically, an eye detection algorithm is applied to face images, assuring the deviation of eye coordinates from expected positions is not significant.
See [NGM19]_ for more details.
Once above script is completed, the data suitable for autoencoder training is located in the folder ``<PATH_TO_STORE_THE_RESULTS>/preprocessed/``. Now the autoencoder can be trained.
The training procedure is explained in the **Convolutional autoencoder** section in the documentation of the ``bob.learn.pytorch`` package.
.. note::
Functionality of ``bob.pad.face`` is used to compute the training data. Install and follow the documentation of ``bob.learn.pytorch`` to train the autoencoders. This functional decoupling helps to avoid the dependency of
``bob.pad.face`` from **PyTorch**.
.. include:: links.rst
2. Fine-tune N AEs on multi-channel data from WMCA (legacy name BATL) database
--grid idiap \ # use grid, only for Idiap users, remove otherwise
--preprocessor video-face-crop-align-bw-ir-d-channels-3x128x128 \ # preprocessor entry point
--sub-directory <PATH_TO_STORE_THE_RESULTS> # define your path here
Once above script is completed, the MC data suitable for autoencoder fine-tuning is located in the folder ``<PATH_TO_STORE_THE_RESULTS>/preprocessed/``.
Now the autoencoder can be fine-tuned. Again, the fine-tuning procedure is explained in the **Convolutional autoencoder** section in the documentation of the ``bob.learn.pytorch`` package.
3. Train an MLP using multi-channel autoencoder latent embeddings from WMCA
Once auto-encoders are pre-trained and fine-tuned, the latent embeddings can be computed passing the multi-channel (MC) BW-NIR-D images from the WMCA database through the encoder, see [NGM19]_ for more details. These latent embeddings (feature vectors) are next used to train an MLP classifying input MC samples into bona-fide or attack classes.
The first step to be done is the registration of an extractor computing latent embeddings. To do so, a file defining an instance of **MultiNetPatchExtractor** class must be created:
.. code-block:: sh
from bob.ip.pytorch_extractor import MultiNetPatchExtractor
from bob.bio.video.utils import FrameSelector
from bob.bio.video.extractor import Wrapper
from torchvision import transforms
from bob.learn.pytorch.architectures import ConvAutoencoder
Suppose, above configuration file is located in ``bob.pad.face`` package in the following location: ``bob/pad/face/config/extractor/multi_net_patch_extractor.py``. Then it can be registered in ``setup.py`` by adding the following string to the list of registered extractors ``bob.pad.extractor``:
--preprocessor video-face-crop-align-bw-ir-d-channels-3x128x128-vect \ # entry point defining preprocessor
--extractor multi-net-patch-extractor \ # entry point defining extractor
--skip-projector-training --skip-projection --skip-score-computation --allow-missing-files \ # execute preprocessing and extraction only
--grid idiap \ # use grid, for Idiap users only, remove otherwise
--sub-directory <PATH_TO_STORE_THE_RESULTS> # define your path here
.. note::
Make sure the ``bob.learn.pytorch`` and ``bob.ip.pytorch_extractor`` packages are installed before running above command.
Once above script is completed, the MC latent encodings to be used for MLP training are located in the folder ``<PATH_TO_STORE_THE_RESULTS>/extracted/``.
Again, the training procedure is explained in the **MLP** section in the documentation of the ``bob.learn.pytorch`` package.
In this section, we briefly describe our work made for face
presentation attack detection using the blood volume pulse,
inferred from remote photoplesthymograpy.
The basic idea here is to retrieve the pulse signals from
face video sequences, to derive features from their frequency
spectrum and then to learn a classifier to discriminate
between *bonafide* attempts from presentation attacks.
For this purpose, we describe both :py:class:`bob.bio.base.preprocessor.Preprocessor` and
:py:class:`bob.bio.base.extractor.Extractor` specifically dedicated to this task.
Preprocessors: Pulse Extraction
-------------------------------
Preprocessors basically extract pulse signals from face video
sequences. They heavily rely on what has been done in `bob.rppg.base`
so you may want to have a look at `its documentation <https://www.idiap.ch/software/bob/docs/bob/bob.rppg.base/master/index.html>`_.
In this package, 4 preprocessors have been implemented:
1. :py:class:`bob.pad.face.preprocessor.LiPulseExtraction` described in [Li_ICPR_2016]_.
2. :py:class:`bob.pad.face.preprocessor.Chrom` described in [CHROM]_.
3. :py:class:`bob.pad.face.preprocessor.SSR` described in [SSR]_.
4. :py:class:`bob.pad.face.preprocessor.PPGSecure` described in [NOWARA]_.
Extractors: Features from Pulses
--------------------------------
Extractors compute and retrieve features from the pulse signal. All
implemented extractors act on the frequency spectrum of the pulse signal.
In this package, 3 extractors have been implemented:
1. :py:class:`bob.pad.face.extractor.LiSpectralFeatures` described in [Li_ICPR_2016]_.
2. :py:class:`bob.pad.face.extractor.PPGSecure` described in [NOWARA]_.
3. :py:class:`bob.pad.face.extractor.LTSS` described in [LTSS]_.
References
----------
.. [Li_ICPR_2016] *X. Li, J, Komulainen, G. Zhao, P-C Yuen and M. Pietikäinen*
**Generalized face anti-spoofing by detecting pulse from face videos**,
Intl Conf on Pattern Recognition (ICPR), 2016
.. [CHROM] *de Haan, G. & Jeanne, V*. **Robust Pulse Rate from Chrominance based rPPG**, IEEE Transactions on Biomedical Engineering, 2013. `pdf <http://www.es.ele.tue.nl/~dehaan/pdf/169_ChrominanceBasedPPG.pdf>`__
.. [SSR] *Wang, W., Stuijk, S. and de Haan, G*. **A Novel Algorithm for Remote Photoplesthymograpy: Spatial Subspace Rotation**, IEEE Trans. On Biomedical Engineering, 2015
.. [NOWARA] *E. M. Nowara, A. Sabharwal, A. Veeraraghavan*. **PPGSecure: Biometric Presentation Attack Detection Using Photopletysmograms**, IEEE International Conference on Automatic Face & Gesture Recognition, 2017
.. [LTSS] *H .Muckenhirn, P. Korshunov, M. Magimai-Doss, S Marcel*. **Long-Term Spectral Statistics for Voice Presentation Attack Detection**, IEEE Trans. On Audio, Speech and Language Processing, 2017
