Commit 748a9676 authored by Yannick DAYER's avatar Yannick DAYER

[doc] More introduction on PAD and vanilla-pad

parent 0f68a1b3
...@@ -9,44 +9,46 @@ ...@@ -9,44 +9,46 @@
Introduction to presentation attack detection Introduction to presentation attack detection
============================================= =============================================
.. todo::
Introduce PAD:
- What it is
- Why
- How
Look at bob.bio.base
Presentation Attack Detection, or PAD, is a branch of biometrics aiming at detecting an attempt to dupe a biometric recognition system by modifying the sample presented to the sensor. Presentation Attack Detection, or PAD, is a branch of biometrics aiming at detecting an attempt to dupe a biometric recognition system by modifying the sample presented to the sensor.
The goal of PAD is to develop countermeasures to presentation attacks that are able to detect wether a biometric sample is a `bonafide` sample, or a presentation attack. The goal of PAD is to develop countermeasures to presentation attacks that are able to detect wether a biometric sample is a `bonafide` sample, or a presentation attack.
For an introduction to biometrics, take a look at the :ref:`documentation of bob.bio.base <bob.bio.base.biometrics_intro>`. For an introduction to biometrics, take a look at the :ref:`documentation of bob.bio.base <bob.bio.base.biometrics_intro>`.
The following introduction to PAD is inspired by chapters 2.4 and 2.6 of [mohammadi2020trustworthy]. The following introduction to PAD is inspired by chapters 2.4 and 2.6 of [mohammadi2020trustworthy]_, and from [marcel2014handbook]_.
Presentation attack Presentation attack
=================== ===================
Biometric recognition systems contain different points of attack. Attacks on certain points are either called direct or indirect attacks. Biometric recognition systems contain different points of attack. Attacks on certain points are either called direct or indirect attacks.
An indirect attack would consist of modifying data after the capture, in any of the steps between the capture and the decision stages. To prevent such attacks is relevant of classical cyber security, hardware and data protection. An indirect attack would consist of modifying data after the capture, in any of the steps between the capture and the decision stages. To prevent such attacks is relevant of classical cyber security, hardware and data protection.
Presentation attacks (PA), on the other hand, are the only direct attacks that can be performed on a biometric system, and countering those attacks is relevant to biometrics. Presentation Attacks (PA), on the other hand, are the only direct attacks that can be performed on a biometric system, and countering those attacks is relevant to biometrics.
For a face recognition system, for example, one of the possible presentation attack would be to wear a mask resembling another individual so that the system identifies the attacker as that other person. For a face recognition system, for example, one of the possible presentation attack would be to wear a mask resembling another individual so that the system identifies the attacker as that other person.
New PAI (Presentation Attack Instrument) can be developed to counteract the countermeasures put in place in the first place, so the field is in constant evolution, to adapt to new threats and try to anticipate them.
Presentation attack detection Presentation attack detection
============================= =============================
A PAD system works much like a biometric recognition system, but with the ability to identify and reject a sample if it is detected as an attack. A PAD system works much like a biometric recognition system, but with the expected ability to identify and reject a sample if it is detected as an attack.
This means that multiple cases are possible and should be detected by a biometric system with PAD: This means that multiple cases are possible and should be detected by a biometric system with PAD:
- A Registered subject presents itself, the captured sample is called **Genuine** sample, and should be accepted by the system (positive), - A Registered subject presents itself, the captured sample is called **Genuine** sample, and should be accepted by the system (positive),
- An Attacker presents itself without trying to pass for another subject, the sample is categorized as **zero effort impostor** sample, and should be rejected by the system (negative), - An Attacker presents itself without trying to pass for another subject, the sample is categorized as **ZEI** (**Zero Effort Impostor**) sample, and should be rejected by the system (negative),
- And the special case in PAD versus "standard" biometric systems: an Attacker uses a `presentation attack instrument` (`PAI`) to pass as a genuine subject. This is a **PA** sample, and should be rejected (negative). - And the special case in PAD versus "standard" biometric systems: an Attacker uses a `Presentation Attack Instrument` (`PAI`) to pass as a genuine subject. This is a **PA** (**Presentation Attack**) sample, and should be rejected (negative).
The term 'bona fide' is used for biometric samples presented without intention to change their identity (Genuine samples and ZEI samples).
The term 'bona fide' is used for biometric samples presented without intention to change their identity (Genuine samples and zero effort impostor samples). .. figure:: img/pad-classes.png
:figwidth: 75%
:align: center
:alt: Four different samples organized to display the different classes of PAD.
Categorization of samples in terms of biometric recognition and PAD systems.
A PAD system makes the distinction between the left samples (`bona-fide`, positives) and the right samples (presentation attack, negatives).
A biometric recognition system, genuine samples are the positives, and both types of impostors are the negatives.
Typical implementations of PAD Typical implementations of PAD
...@@ -61,10 +63,72 @@ PAD for face recognition is the most advanced in this field, face PAD systems ca ...@@ -61,10 +63,72 @@ PAD for face recognition is the most advanced in this field, face PAD systems ca
PAD system using a frame-based approach on visible light with no user interaction are the least robust but are more developed, as they can be easily integrated with existing biometric systems. PAD system using a frame-based approach on visible light with no user interaction are the least robust but are more developed, as they can be easily integrated with existing biometric systems.
Evaluation of PAD systems
=========================
To evaluate a biometric system with PAD, a set of samples is fed to the system. Each samples is scored, and a post processing step is used to analyse those scores.
Licit scenario
--------------
When no PA samples are in the input set (only Genuine and ZEI samples), the situation is the same as a simple biometric experiment and is called `licit` scenario. See :ref:`biometric introduction<bob.bio.base.biometrics_intro>`.
Spoof scenario
--------------
If no ZEI samples are present in the set (only Genuine and PA samples), the evaluation of a PAD system is seen as a two classes problem, and the same metrics as in a biometric evaluation can be used to assess its performance, where:
- the False Positive Rate is called IAPMR (Impostor Attack Presentation Match Rate),
- the False Negative Rate is called FNMR (False Non Match Rate),
The ROC and DET can be plotted to represent the performance of the system over a range of operation points.
This two classes case is referred as the `spoof` scenario.
PAD evaluation
--------------
When a mix of Zero Effort Impostor and PA are present in the input set, two possibilities arise.
The bona fide (Genuine and ZEI) samples are treated as `positives` and PA samples are considered `negatives` (This will show the ability of the system to detect PA).
The problem becomes binary, allowing the use of similar metrics as before, albeit with different denominations:
- the False Positive Rate is named APCER (Attack Presentation Classification Error Rate),
- the False Negative Rate is named BPCER (Bona fide Presentation Classification Error Rate),
- the Half Total Error Rate is named ACER (Average Classification Error Rate).
The ZEI and PA samples can also be considered two separate negative classes, leading to a ternary classification with one positive class (genuine samples) and two distinct negative classes: ZEI and PA.
The EPS (Expected Performance and Spoofability) framework was introduced to assess the reliability of a biometric system with PAD by defining two parameters determining how much importance is given to each class of samples:
- ω represents the importance of the PA scores with respect to the ZEI scores.
- β represents the importance of the negative classes (PA and ZEI scores) relative to the positive class (Genuine).
From the scores and those two parameters, the following metrics can be measured:
- The weighted error rate for the two negative classes (IAPMR for the PA scores and FMR for the ZEI scores):
:math:`\text{FAR}_\omega = \omega \cdot \text{IAPMR} + (1-\omega) \cdot \text{FMR}`
- The weighted error rate between the previously defined :math:`\text{FAR}_\omega` and the :math:`\text{FNMR}` (between Genuine and both negatives), computed as:
:math:`\text{WER}_{\omega.\beta} = \beta \cdot \text{FAR}_\omega + (1-\beta) \cdot \text{FNMR}`
ω and β are chosen by minimizing :math:`\text{WER}_{\omega,\beta}` on the `development set` scores. Then by using those values, the :math:`\text{WER}_{\omega,\beta}` is computed on the `evaluation set` scores.
.. note:: :math:`\text{HTER}_\omega` is also defined when :math:`\beta = 0.5` : :math:`\text{HTER}_\omega = {\text{FAR}_\omega + \text{FNMR} \over 2}`
The EPSC curve can be plotted to assess the performance of the system on various ω, by fixing β. It plots the error rate :math:`\text{WER}_{\omega,\beta}` against the weight ω.
The EPSC can also be in 3D if β is not fixed, showing the :math:`\text{WER}_{\omega,\beta}` against both weights ω and β.
References References
========== ==========
.. [mohammadi2020trustworthy] * Mohammadi Amir **Trustworthy Face Recognition: Improving Generalization of Deep Face Presentation Attack Detection**, 2020, EPFL .. [mohammadi2020trustworthy] * Mohammadi Amir **Trustworthy Face Recognition: Improving Generalization of Deep Face Presentation Attack Detection**, 2020, EPFL
.. [marcel2014handbook] * Marcel, Sébastien and Nixon, Mark S and Li, Stan Z **Handbook of biometric anti-spoofing**, 2014, Springer
.. include:: links.rst .. include:: links.rst
...@@ -22,6 +22,8 @@ ...@@ -22,6 +22,8 @@
To easily run experiments in PAD, we offer a generic command called ``bob pad pipelines``. To easily run experiments in PAD, we offer a generic command called ``bob pad pipelines``.
Such CLI command is an entry point to several pipelines, and this documentation will focus on the one called **vanilla-pad**. Such CLI command is an entry point to several pipelines, and this documentation will focus on the one called **vanilla-pad**.
The following will introduce how a simple experiment can be run with this tool, from the samples data to a set of metrics and plots, as defined in :ref:`bob.pad.base.pad_intro`.
Running a biometric experiment with vanilla-pad Running a biometric experiment with vanilla-pad
=============================================== ===============================================
...@@ -62,14 +64,13 @@ Building your own Vanilla PAD pipeline ...@@ -62,14 +64,13 @@ Building your own Vanilla PAD pipeline
The Vanilla PAD pipeline is the backbone of any experiment in this library. It is composed of: The Vanilla PAD pipeline is the backbone of any experiment in this library. It is composed of:
- Transformers: One or multiple instances in series of :py:class:`sklearn.base.BaseEstimator` and :py:class:`sklearn.base.TransformerMixin`. A transformer takes a sample as input applies a modification on it and outputs the resulting sample. - Transformers: One or multiple instances in series of :py:class:`sklearn.base.BaseEstimator` and :py:class:`sklearn.base.TransformerMixin`. A transformer takes a sample as input applies a modification on it and outputs the resulting sample. A transformer can be trained before being used.
A transformer can be trained before using it.
- A classifier: Instance of - A classifier: Instance of
.. todo:: .. todo::
instance of the classifier Add the Instance of the classifier
Transformers Transformers
...@@ -78,17 +79,141 @@ Transformers ...@@ -78,17 +79,141 @@ Transformers
A Transformer is an class that implements the fit and transform methods, which allow the application of an operation on a sample of data. A Transformer is an class that implements the fit and transform methods, which allow the application of an operation on a sample of data.
For more details, see :ref:`bob.bio.base.transformer`. For more details, see :ref:`bob.bio.base.transformer`.
.. todo::
Explain where they are / how to build one
Classifier Classifier
---------- ----------
A Classifier is the final process of a Vanilla PAD pipeline. Its goal is to decide if a transformed sample given as input is originating from a genuine sample or if an impostor is trying to be recognized as someone else. A Classifier is the final process of a Vanilla PAD pipeline. Its goal is to decide if a transformed sample given as input is originating from a genuine sample or if an impostor is trying to be recognized as someone else.
.. todo::
Explain where they are / how to build one
Running an experiment Running an experiment
===================== =====================
Two part of an experiment have to be executed: Two part of an experiment have to be executed:
- Fit: - **Fit**: labeled data is fed to the system to train the algorithm to recognize attacks and licit proprieties.
- Predict: assessing a series of test samples for authenticity. - **Predict**: assessing a series of test samples for authenticity, generating a score for each one.
\ No newline at end of file
.. todo::
Examples
Evaluation
----------
Once the scores are generated for each classes and groups, the evaluation tools can be used to assess the performance of the system, by either drawing plots or computing metrics values at specific operation points.
Generally, the operation thresholds are computed on a specific set (development set or `dev`). Then those threshold values are used to compute the system error rates on a separate set (evaluation set or `eval`).
To retrieve the most common metrics values for a spoofing scenario experiment, run the following command:
.. code-block:: none
$ bob pad metrics -e scores-{dev,eval} --legends ExpA
Threshold of 11.639561 selected with the bpcer20 criteria
====== ======================== ===================
ExpA Development scores-dev Eval. scores-eval
====== ======================== ===================
APCER 5.0% 5.0%
BPCER 100.0% 100.0%
ACER 52.5% 52.5%
====== ======================== ===================
Threshold of 3.969103 selected with the eer criteria
====== ======================== ===================
ExpA Development scores-dev Eval. scores-eval
====== ======================== ===================
APCER 100.0% 100.0%
BPCER 100.0% 100.0%
ACER 100.0% 100.0%
====== ======================== ===================
Threshold of -0.870550 selected with the min-hter criteria
====== ======================== ===================
ExpA Development scores-dev Eval. scores-eval
====== ======================== ===================
APCER 100.0% 100.0%
BPCER 19.5% 19.5%
ACER 59.7% 59.7%
====== ======================== ===================
.. note::
When evaluation scores are provided, the ``-e`` option (``--eval``) must be passed.
See metrics --help for further options.
Plots
-----
Customizable plotting commands are available in the :py:mod:`bob.pad.base` module.
They take a list of development and/or evaluation files and generate a single PDF
file containing the plots.
Available plots for a spoofing scenario are:
* ``hist`` (Bona fida and PA histograms along with threshold criterion)
* ``epc`` (expected performance curve)
* ``gen`` (Generate random scores)
* ``roc`` (receiver operating characteristic)
* ``det`` (detection error trade-off)
* ``evaluate`` (Summarize all the above commands in one call)
Available plots for vulnerability analysis are:
* ``hist`` (Vulnerability analysis distributions)
* ``epc`` (expected performance curve)
* ``gen`` (Generate random scores)
* ``roc`` (receiver operating characteristic)
* ``det`` (detection error trade-off)
* ``epsc`` (expected performance spoofing curve)
* ``fmr_iapmr`` (Plot FMR vs IAPMR)
* ``evaluate`` (Summarize all the above commands in one call)
Use the ``--help`` option on the above-cited commands to find-out about more
options.
For example, to generate a EPC curve from development and evaluation datasets:
.. code-block:: sh
$ bob pad epc -e -o 'my_epc.pdf' scores-{dev,eval}
where `my_epc.pdf` will contain EPC curves for all the experiments.
Vulnerability commands require licit and spoof development and evaluation
datasets. Far example, to generate EPSC curve:
.. code-block:: sh
$ bob vuln epsc -e .../{licit,spoof}/scores-{dev,eval}
.. note::
IAPMR curve can be plotted along with EPC and EPSC using option
``--iapmr``. 3D EPSC can be generated using the ``--three-d``. See metrics
--help for further options.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment