experiments.rst

Running Biometric Recognition Experiments

Now, you are almost ready to run your first biometric recognition experiment. Just a little bit of theory, and then: off we go.

Structure of a Biometric Recognition Experiment

Each biometric recognition experiment that is run with bob.bio is divided into several steps. The steps are:

1. Data preprocessing: Raw data is preprocessed, e.g., for face recognition, faces are detected, images are aligned and photometrically enhanced.
2. Feature extractor training: Feature extraction parameters are learned.
3. Feature extraction: Features are extracted from the preprocessed data.
4. Feature projector training: Parameters of a subspace-projection of the features are learned.
5. Feature projection: The extracted features are projected into a subspace.
6. Model enroller training: The ways how to enroll models from extracted or projected features is learned.
7. Model enrollment: One model is enrolled from the features of one or more images.
8. Scoring: The verification scores between various models and probe features are computed.
9. Evaluation: The computed scores are evaluated and curves are plotted.

These 9 steps are divided into four distinct groups, which are discussed in more detail later:

• Preprocessing (only step 1)
• Feature extraction (steps 2 and 3)
• Biometric recognition (steps 4 to 8)
• Evaluation (step 9)

The communication between two steps is file-based, usually using a binary HDF5_ interface, which is implemented in the :py:class:`bob.io.base.HDF5File` class. The output of one step usually serves as the input of the subsequent step(s). Depending on the algorithm, some of the steps are not applicable/available. E.g. most of the feature extractors do not need a special training step, or some algorithms do not require a subspace projection. In these cases, the according steps are skipped. bob.bio takes care that always the correct files are forwarded to the subsequent steps.

Running Experiments (part I)

To run an experiment, we provide a generic script ./bin/verify.py, which is highly parametrizable. To get a complete list of command line options, please run:

$ ./bin/verify.py --help

Whoops, that's a lot of options. But, no worries, most of them have proper default values.

Note

Sometimes, command line options have a long version starting with -- and a short one starting with a single -. In this section, only the long names of the arguments are listed, please refer to ./bin/verify.py --help (or short: ./bin/faceverify.py -h) for the abbreviations.

There are five command line options, which are required and sufficient to define the complete biometric recognition experiment. These five options are:

• --database: The database to run the experiments on
• --preprocessor: The data preprocessor
• --extractor: The feature extractor
• --algorithm: The recognition algorithm
• --sub-directory: A descriptive name for your experiment, which will serve as a sub-directory

The first four parameters, i.e., the database, the preprocessor, the extractor and the algorithm can be specified in several different ways. For the start, we will use only the registered :ref:`Resources <bob.bio.base.resources>`. These resources define the source code that will be used to compute the experiments, as well as all the meta-parameters of the algorithms (which we will call the configuration). To get a list of registered resources, please call:

$ ./bin/resources.py

Each package in bob.bio defines its own resources, and the printed list of registered resources differs according to the installed packages. If only bob.bio.base is installed, no databases and only one preprocessor will be listed. To see more details about the resources, i.e., the full constructor call fo the respective class, use the --details (or shortly -d) option, and to sub-select only specific types of resources, use the --types (or -t) option:

$ ./bin/resources.py -dt algorithm

Note

You will also find some grid resources being listed. These type of resources will be explained :ref:`later <running_in_parallel>`.

Before going into :ref:`more details about the configurations <running_part_2>`, we will provide information about running default experiments.

One command line option, which is not required, but recommended, is the --verbose option. By default, the algorithms are set up to execute quietly, and only errors are reported. To change this behavior, you can use the --verbose option several times to increase the verbosity level to show:

1. Warning messages
2. Informative messages
3. Debug messages

When running experiments, my personal preference is verbose level 2, which can be enabled by --verbose --verbose, or using the short version: -vv. So, a typical biometric recognition experiment (in this case, face recognition) could look something like:

$ ./bin/verify.py --database mobio-image --preprocessor face-crop-eyes --extractor linearize --algorithm pca --sub-directory pca-experiment -vv

Note

To be able to run exactly the command line from above, it requires to have :ref:`bob.bio.face <bob.bio.face>` installed.

Before running an experiment, it is recommended to add the --dry-run option, so that it will only print, which steps would be executed, without actually executing them, and make sure that everything works as expected.

The final result of the experiment will be one (or more) score file(s). Usually, they will be called something like scores-dev. By default, you can find them in a sub-directory the result directory, but you can change this option using the --result-directory command line option.

Note

At Idiap_, the default result directory differs, see ./bin/verify.py --help for your directory.

Evaluating Experiments

After the experiment has finished successfully, one or more text file containing all the scores are written.

To evaluate the experiment, you can use the generic ./bin/evaluate.py script, which has properties for all prevalent evaluation types, such as CMC, ROC and DET plots, as well as computing recognition rates, EER/HTER, Cllr and minDCF. Additionally, a combination of different algorithms can be plotted into the same files. Just specify all the score files that you want to evaluate using the --dev-files option, and possible legends for the plots (in the same order) using the --legends option, and the according plots will be generated. For example, to create a ROC curve for the experiment above, use:

$ ./bin/evaluate.py --dev-files results/pca-experiment/male/nonorm/scores-dev --legend MOBIO --roc MOBIO_MALE_ROC.pdf -vv

Please note that there exists another file called Experiment.info inside the result directory. This file is a pure text file and contains the complete configuration of the experiment. With this configuration it is possible to inspect all default parameters of the algorithms, and even to re-run the exact same experiment.

Running in Parallel

One important property of the ./bin/verify.py script is that it can run in parallel, using either several threads on the local machine, or an SGE grid. To achieve that, bob.bio is well-integrated with our SGE grid toolkit GridTK_, which we have selected as a python package in the :ref:`Installation <bob.bio.base.installation>` section. The ./bin/verify.py script can submit jobs either to the SGE grid, or to a local scheduler, keeping track of dependencies between the jobs.