diff --git a/doc/1_mc_ae_face_pad.rst b/doc/1_mc_ae_face_pad.rst new file mode 100644 index 0000000000000000000000000000000000000000..a130a85a83290c374a87e66054d554db012fa9fa --- /dev/null +++ b/doc/1_mc_ae_face_pad.rst @@ -0,0 +1,83 @@ + + +.. _bob.paper.mcae.icb2019.1_mc_ae_face_pad: + + +=============================================================== + Multi-channel face PAD using 1 autoencoder (baseline system) +=============================================================== + +This section explains how to run a complete face PAD experiment using multi-channel autoencoder-based face PAD system, as well as a training work-flow. + +The system discussed in this section is introduced the following publication [NGM19]_. It is **strongly recommended** to check the publication for better understanding +of the described work-flow. + +.. note:: + + Below discussions rely on the fact, that two databases, CelebA and WMCA, are downloaded and installed in your system. Please refer to :ref:`bob.pad.face.baselines` section in the documentation of ``bob.pad.face`` package for more details on how to run the face PAD experiments and setup the databases. + + +Training multi-channel autoencoder-based face PAD system. +---------------------------------------------------------------- + +The training procedure of multi-channel face PAD system using **one** autoencoder is thoroughly discussed in the documentation of ``bob.pad.face`` package, section on :ref:`bob.pad.face.mc_autoencoder_pad`, thus it is not repeated in current instructions. + + +Running face PAD Experiments +------------------------------ + +After the training procedure is completed, one can run a complete PAD experiment. The command to run **two** PAD experiments on the multi-channel, BW-NIR-D, data of WMCA: + +.. code-block:: sh + + ./bin/spoof.py \ # spoof.py is used to run the experiment + batl-db-rgb-ir-d-grandtest \ # WMCA database instance allowing to loading RGB-NIR-D channels + mc-pad-bw-nir-d-1-ae-celeba-pretrain \ # configuration defining Preprocessor, Extractor, and Algorithm instances + --allow-missing-files \ # don't stop the execution if some files are missing + --grid idiap \ # use grid, only for Idiap users, REMOVE otherwise + --sub-directory <FOLDER_TO_SAVE_THE_RESULTS_1> # define your path here + + ./bin/spoof.py \ + batl-db-rgb-ir-d-grandtest \ + mc-pad-bw-nir-d-1-ae-batl-3-layers-finetune \ + --allow-missing-files \ + --grid idiap \ + --sub-directory <FOLDER_TO_SAVE_THE_RESULTS_2> + +The **first** experiment corresponds to the PAD system composed of: preprocessor extracting 1 MC facial patch (entire face), extractor with 1 autoencoder trained on CelebA only, and an MLP algorithm trained on MC latent embeddings. + +The **second** experiment corresponds to the PAD system composed of: preprocessor extracting 1 MC facial patch (entire face), extractor with 1 autoencoder trained on CelebA and fine-tuned on the WMCA, and an MLP algorithm trained on MC latent embeddings. + +.. note:: + + Entry points ``mc-pad-bw-nir-d-1-ae-celeba-pretrain`` and ``mc-pad-bw-nir-d-1-ae-batl-3-layers-finetune`` are pointing to corresponding configuration files, which initialize autoencoders and MLP using pre-trained models distributed with the package. If you want to use the models computed in the previous steps, revise the configuration files pointing to models from your system. + +After an execution of above scripts is completed the score files will be located in the folders: ``<FOLDER_TO_SAVE_THE_RESULTS_1>/grandtest-color*infrared*depth-10/scores/`` and ``<FOLDER_TO_SAVE_THE_RESULTS_2>/grandtest-color*infrared*depth-10/scores/``. + + +Evaluating results of face PAD Experiments +--------------------------------------------- + +Assuming the score files, for **two** experiments discussed above, are located in the folders ``<FOLDER_TO_SAVE_THE_RESULTS_1>/grandtest-color*infrared*depth-10/scores/`` and ``<FOLDER_TO_SAVE_THE_RESULTS_2>/grandtest-color*infrared*depth-10/scores/``, one can use the following command to evaluate the results: + +.. code-block:: sh + + ./bin/bob bio evaluate -e -v -l '<SAVE_FOLDER>/metrix.txt' \ + -o '<SAVE_FOLDER>/curves.pdf' \ + -lg "1 MC patch + CelebA AE + 10 ReLU",\ + "1 MC patch + CelebA & BATL 3-layer-tune AE + 10 ReLU" \ + <FOLDER_TO_SAVE_THE_RESULTS_1>/grandtest-color*infrared*depth-10/scores/scores-dev \ + <FOLDER_TO_SAVE_THE_RESULTS_1>/grandtest-color*infrared*depth-10/scores/scores-eval \ + <FOLDER_TO_SAVE_THE_RESULTS_2>/grandtest-color*infrared*depth-10/scores/scores-dev \ + <FOLDER_TO_SAVE_THE_RESULTS_2>/grandtest-color*infrared*depth-10/scores/scores-eval + +The files **metrix.txt** and **curves.pdf**, containing error rates and evaluation curves, are saved to ``<SAVE_FOLDER>`` location. Inspecting the **curves.pdf**, one can find the curves similar to this one: + +.. figure:: img/roc_1_ae_pad_system.png + :align: center + + ROC curves for PAD system using 1 AE for MC face, and an MLP classifier. For **evaluation** set of WMCA, **grandtest-10** protocol. + +.. include:: links.rst + + diff --git a/doc/9_mc_ae_face_pad.rst b/doc/9_mc_ae_face_pad.rst index 8c555e95f864cf097917948516e9b3ee64cc988d..0d9df52562e0479766b1325c27e383d13a635c40 100644 --- a/doc/9_mc_ae_face_pad.rst +++ b/doc/9_mc_ae_face_pad.rst @@ -3,9 +3,9 @@ .. _bob.paper.mcae.icb2019.9_mc_ae_face_pad: -============================================= - Multi-channel face PAD using 9 autoencoders -============================================= +============================================================ + Multi-channel face PAD using 9 autoencoders (best system) +============================================================ This section explains how to run a complete face PAD experiment using multi-channel autoencoder-based face PAD system, as well as a training work-flow. @@ -605,7 +605,7 @@ After the training procedure, discussed above, is completed, one can run a compl mc-pad-bw-nir-d-9-ae-batl-3-layers-finetune \ --allow-missing-files \ --grid idiap \ - --sub-directory <FOLDER_TO_SAVE_THE_RESULTS_1> + --sub-directory <FOLDER_TO_SAVE_THE_RESULTS_2> The **first** experiment corresponds to the PAD system composed of: preprocessor extracting 9 MC facial patches, extractor with 9 autoencoders trained on CelebA only, and an MLP algorithm trained on MC latent embeddings. diff --git a/doc/img/roc_1_ae_pad_system.png b/doc/img/roc_1_ae_pad_system.png new file mode 100644 index 0000000000000000000000000000000000000000..9dd2360143829fb54f4357a6a04ce6ff810b052b Binary files /dev/null and b/doc/img/roc_1_ae_pad_system.png differ diff --git a/doc/index.rst b/doc/index.rst index 4c129c4a110277c0469eab2f70cc23f1a3e759c0..a3f469c02b4e3361dc55a955d7151bfe9584dc2f 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -26,6 +26,7 @@ User guide :maxdepth: 2 9_mc_ae_face_pad + 1_mc_ae_face_pad references