experiments.rst

Running Presentation Attack Detection Experiments

Now, you are almost ready to run presentation attack detection (PAD) experiment.

Structure of a PAD Experiment

Each biometric recognition experiment that is run with bob.pad is divided into the following several steps:

1. Data preprocessing: Raw data is preprocessed, e.g., for speech, voice activity is detected.
2. Feature extraction: Features are extracted from the preprocessed data.
3. Feature projector training: Models of genuine data and attacks are learnt.
4. Feature projection: The extracted features are projected into corresponding subspaces.
5. Scoring: The spoofing scores for genuine data and attacks are computed.
6. Evaluation: The computed scores are evaluated and curves are plotted.

These 6 steps are divided into four distinct groups:

• Preprocessing (step 1)
• Feature extraction (step 2)
• Attack detection (steps 3 to 5)
• Evaluation (step 6)

The communication between two steps is file-based, usually using a binary HDF5_ interface, which is implemented in the :py:class:`bob.io.base.HDF5File` class. The output of one step usually serves as the input of the subsequent step(s). Depending on the algorithm, some of the steps are not applicable/available. bob.pad takes care that always the correct files are forwarded to the subsequent steps.

Running Experiments

To run an experiment, we provide a generic script ./bin/spoof.py. To get a complete list of command line options, please run:

$ ./bin/spoof.py --help

Note

Sometimes, command line options have a long version starting with -- and a short one starting with a single -. In this section, only the long names of the arguments are listed, please refer to ./bin/spoof.py --help.

There are five command line options, which are required and sufficient to define the complete biometric recognition experiment. These five options are:

• --database: The database to run the experiments on
• --preprocessor: The data preprocessor
• --extractor: The feature extractor
• --algorithm: The presentation attack detection algorithm
• --sub-directory: A descriptive name for your experiment, which will serve as a sub-directory

The first four parameters, i.e., the database, the preprocessor, the extractor and the algorithm can be specified in several different ways. For the start, we will use only the registered :ref:`Resources <bob.bio.base.resources>`. These resources define the source code that will be used to compute the experiments, as well as all the meta-parameters of the algorithms (which we will call the configuration). To get a list of registered resources, please call:

$ ./bin/resources.py

Each package in bob.pad defines its own resources, and the printed list of registered resources differs according to the installed packages. If only bob.pad.base is installed, no databases and no preprocessors will be listed.

Note

You will also find some grid resources being listed. These type of resources will be explained :ref:`later <running_in_parallel>`.

One command line option, which is not required, but recommended, is the --verbose option. By default, the algorithms are set up to execute quietly, and only errors are reported. To change this behavior, you can use the --verbose option several times to increase the verbosity level to show:

1. Warning messages
2. Informative messages
3. Debug messages

When running experiments, it is a good idea to set verbose level 2, which can be enabled by using the short version: -vv. So, a typical PAD experiment (in this case, attacks detection in speech) would look like the following:

$ ./bin/spoof.py --database <database-name> --preprocessor <preprocessor> --extractor <extractor> --algorithm <algorithm> --sub-directory <folder_name> -vv

Before running an experiment, it is recommended to add the --dry-run option, so that it will only print, which steps would be executed, without actually executing them, and make sure that everything works as expected.

The final result of the experiment will be one (or more) score file(s). Usually, they will be called something like scores-dev-real for genuine data, scores-dev-attack for attacks, and scores-dev for the results combined in one file. By default, you can find them in a sub-directory the result directory, but you can change this option using the --result-directory command line option.

Note

At Idiap_, the default result directory differs, see ./bin/spoof.py --help for your directory.

Evaluating Experiments

After the experiment has finished successfully, one or more text file containing all the scores are written.