Commit 86b8d6e0 authored by Pavel KORSHUNOV's avatar Pavel KORSHUNOV

Merge branch 'corrections' into 'master'

updated docs and evaluation scripts

See merge request !2
parents 41ecab5a 6ae99182
Pipeline #9285 passed with stages
in 9 minutes and 8 seconds
......@@ -72,6 +72,7 @@ def main(command_line_parameters=None):
fid = open(args.out_file, "w")
# fid.write("{\bf Feat.\ type} & & {\bf DEV Threshold} & {\bf DEV EER (\%)} & {\bf EVAL HTER (\%)} \\ \hline \n")
fid.write("Feature & type_attack & type_device & SFAR & SFRR & HTER \\\\ \\hline \n")
for score_type in args.score_types:
for param in args.parameters:
cur_dir = args.directory + "_" + score_type + "_" + param
......@@ -125,10 +126,10 @@ def main(command_line_parameters=None):
if type_attack == 'all':
# tableline = "%s %.3f & %.5f & %.5f \\\\ \\hline \n" % \
# (title, eer_thres, (far+frr)*0.5, hter)
tableline = "%s & all & all & %.3f & %.3f & %.3f \\\\ \\hline \n" % \
tableline = "%s & all & all & %.3f & %.3f & %.3f \\\\ \n" % \
(title, sfrr, sfar, hter)
else:
tableline = "%s & %s & %s & %.3f & %.3f & %.3f \\\\ \\hline \n" % \
tableline = "%s & %s & %s & %.3f & %.3f & %.3f \\\\ \n" % \
(title, type_attack, type_device, sfrr, sfar, hter)
fid.write(tableline)
fid.write(" \hline \n")
......
......@@ -19,10 +19,28 @@ This package is part of the Bob_ toolkit, which allows to reproduce the results
}
Installation
------------
Follow our `installation`_ instructions for installing ``conda`` and core Bob_ packages.
Also, install ``bob.bio.*``, ``bob.pad.*``, and database packages by running::
$ conda install bob.bio.base bob.bio.spear bob.bio.gmm bob.pad.base bob.pad.voice bob.db.avspoof bob.db.asvspoof
Then, using the Python from the installed conda environment, bootstrap and buildout this package::
$ python bootstrap-buildout.py
$ ./bin/buildout
Running all experiments from scratch
------------------------------------
This package contains basic scripts to run Presentation Attack Detection (PAD) speech experiments presented in the paper.
It uses ``bob.bio.*`` and ``bob.pad.*`` packages from Bob framework, as well as, Bob's database interfaces for AVspoof_
and ASVspoof_ databases in ``bob.db.avspoof`` and ``bob.db.asvspoof``. All the above packages are assumed to be
installed using ``conda`` environment or built using ``buildout``.
and ASVspoof_ databases that are defined in ``bob.db.avspoof`` and ``bob.db.asvspoof`` packages. All these packages are
assumed to be installed using ``conda`` environment or built using ``buildout``.
The provided experimental pipeline scripts take several parameters, including:
......@@ -35,34 +53,44 @@ All these steps of the PAD system are given as configuration files.
To run all the experiments, two databases need to be downloaded: AVspoof_ and ASVspoof_.
Once the databases are downloaded, please specify the paths to these databases by creating/editing file ``~/
.bob_bio_databases.txt`` and writing there the following two lines:
Once the databases are downloaded, please specify the paths to these databases by creating/editing file
``~/.bob_bio_databases.txt`` and writing in it the following two lines:
$ [YOUR_AVSPOOF_WAV_DIRECTORY]=/absolute/path/to/avspoof/database/
$ [YOUR_ASVSPOOF_WAV_DIRECTORY]=/absolute/path/to/asvspoof/database/
Now everything is ready to run the experiments. Here is a generic command for training GMM-based PAD system::
By using ``~/.bob_bio_databases.txt`` file, where path placeholders are replaced with real paths, bob's framework
packages can find the data needed for the experiments.
GMM models for ``real`` and ``spoofed`` data are trained as per this generic command::
$ ./bin/train_gmm.py -d DB_NAME -p Preprocessor -e Feature_Extractor -a Classifier -s Folder_Name --groups world --skip-enroller-training -vv --parallel 6
$ ./bin/train_gmm.py -d DB_NAME -p Preprocessor -e Feature_Extractor -a gmm-tomi -s Folder_Name --groups world --skip-enroller-training -vv
This training may take a long time and as the results, it will generate GMM model and write it into ``Projector.hdf5``
file. You can check all possible options by running ``$ ./bin/train_gmm.py --help``.
Here is the generic command to tune the system on developing set and evaluate on the test set::
Here is the generic command for tuning the trained system on developing set and evaluate on the test set::
$ ./bin/spoof.py -d DB_NAME -p Preprocessor -e Feature_Extractor -a gmm --projector-file Projector_spoof.hdf5 -s Folder_Name --groups dev eval --skip-projector-training -vv
$ ./bin/spoof.py -d DB_NAME -p Preprocessor -e Feature_Extractor -a Classifier --projector-file Projector_spoof.hdf5 -s Folder_Name --groups dev eval --skip-projector-training -vv
For example, to train and evaluate a GMM-based PAD system using MFCC-based features for Licit protocol of the ASVspoof database, the following commands need to be run::
For example, to train and evaluate a GMM-based PAD system using MFCC-based features computed for
``licit`` and ``spoof`` protocols of the ASVspoof database, the following commands need to be run::
$ ./bin/train_gmm.py -d asvspoof-licit -p mod-4hz -e mfcc20 -a gmm-tomi -s temp --groups world
--skip-enroller-training -vv --parallel 6
$ ./bin/train_gmm.py -d asvspoof-spoof -p mod-4hz -e mfcc20 -a gmm-tomi -s temp --groups world
--skip-enroller-training -vv --parallel 6
$ ./bin/spoof.py -d asvspoof -p mod-4hz -e mfcc20 -a gmm --projector-file Projector_spoof.hdf5 -s temp
--groups dev eval --skip-projector-training -vv
$ ./bin/train_gmm.py -d asvspoof-licit -p mod-4hz -e mfcc20 -a gmm-tomi -s temp --groups world --projector-file Projector_licit.hdf5 --skip-enroller-training -vv --parallel 6
$ ./bin/train_gmm.py -d asvspoof-spoof -p mod-4hz -e mfcc20 -a gmm-tomi -s temp --groups world --projector-file Projector_spoof.hdf5 --skip-enroller-training -vv --parallel 6
$ ./bin/spoof.py -d asvspoof -p mod-4hz -e mfcc20 -a gmm --projector-file Projector_spoof.hdf5 -s temp --groups dev eval --skip-projector-training -vv
Then, using the obtained scores, error rates can be computed and DET curves plotted using the following script::
Here, ``./bin/train_gmm.py`` produces two GMM models, one for ``licit`` protocol (real data only) and one for ``spoof``
protocol (spoofed data only). Then, ``./bin/spoof.py`` is used to project all data from ``dev`` and ``eval`` sets onto
these two GMM models and compute corresponding scores.
Once the scores are obtained, error rates and DET curves can be computed using the following script::
$ ./bin/plot_pad_results.py -t scores_path/dev-attack -d scores_path/dev-real -f scores_path/eval-attack -e scores_path/eval-real -o plots"
Also, it is possible to reproduce the experiments presented in the paper using the following bash scripts that run for all PAD systems used::
Also, it is possible to reproduce the experiments presented in the paper using the following bash scripts that run for all
PAD systems used in the paper (note that these scripts assume SGE grid to be available and will take a few days on 50
parallel machines)::
$ ./train_gmms.sh avspoof 20 # train for AVspoof database
$ ./train_gmms.sh asvspoof 20 # train for ASVspoof database
......@@ -80,7 +108,8 @@ If you want to avoid training all PAD systems and computing scores, we are provi
$ tar -xzvf interspeech_2016.tar.gz
$ ./evaluate_scores.sh # compute error rates and plot the DET curves for each PAD system
The script will create folders for each different PAD system (it contains computed error rates and DET curves) and one ``stats.txt`` file with error rates from all systems in one LaTeX table.
The script will create folders for each different PAD system (it contains computed error rates and DET curves)
and one ``stats.txt`` file with error rates from all systems in one LaTeX table.
To plot combined DET curves for different systems as per Figure 2 of the paper, the following script can be run::
......@@ -88,13 +117,8 @@ To plot combined DET curves for different systems as per Figure 2 of the paper,
This script will plot several DET curves in a single PDF file inside the folder ``plots_compare_pads``.
Installation
------------
To install this package -- alone or together with other `Packages of Bob <https://github.com/idiap/bob/wiki/Packages>`_ -- please read the `Installation Instructions <https://github.com/idiap/bob/wiki/Installation>`_.
For Bob_ to be able to work properly, some dependent packages are required to be installed.
Please make sure that you have read the `Dependencies <https://github.com/idiap/bob/wiki/Dependencies>`_ for your operating system.
.. _bob: https://www.idiap.ch/software/bob
.. _installation: https://www.idiap.ch/software/bob/install
.. _AVspoof: https://www.idiap.ch/dataset/avspoof
.. _ASVspoof: http://datashare.is.ed.ac.uk/handle/10283/853
......@@ -5,22 +5,25 @@ prettytitles=("SCFC(20) $\Delta\Delta^2$, GMM512" "RFCC(20) $\Delta\Delta^2$, GM
# types of experiments
postfixes=(avspoof_logic avspoof_phys avspoof_asvspoof2logic avspoof_asvspoof2phys asvspoof_avspoof2pad avspoof_logic2phys)
prettypostfixes=(", ASVspoof-LA" ", ASVspoof-PA" ", ASVspoof (train,dev) on AVspoof-LA (eval)" ", ASVspoof (train,dev) on AVspoof-PA (eval)" ", AVspoof-LA (train,dev) on ASVspoof (eval)" ", AVspoof-LA (train,dev) on AVspoof-PA (eval)")
dev_scores=(asvspoof_pad avspoof_logic avspoof_phys asvspoof_pad asvspoof_pad avspoof_logic avspoof_logic)
prettypostfixes=(", ASVspoof " ", AVspoof-LA" ", AVspoof-PA" ", ASVspoof (train,dev) on AVspoof-LA (eval)" ", ASVspoof (train,dev) on AVspoof-PA (eval)" ", AVspoof-LA (train,dev) on ASVspoof (eval)" ", AVspoof-LA (train,dev) on AVspoof-PA (eval)")
eval_scores=(asvspoof_pad avspoof_logic avspoof_phys avspoof_asvspoof2logic avspoof_asvspoof2phys asvspoof_avspoof2pad avspoof_logic2phys)
indxfixes=0
for p in "${postfixes[@]}"; do
for dd in "${dev_scores[@]}"; do
ed=${eval_scores[indxfixes]}
indxfeat=0
for f in "${features[@]}"; do
echo "=============================================================="
echo "Evaluate scores for '${f}' features of ${database} database"
echo "=============================================================="
scorepath_dev="./scores/${p}/${f}_20/scores-"
scorepath="./scores/${p}/${f}_20/scores-"
scorepath_dev="./scores/${dd}/${f}_20/scores-dev"
scorepath_eval="./scores/${ed}/${f}_20/scores-eval"
plotnames+=("${p}_${f}")
command="-n 10 -m 60 -t ${scorepath_dev}dev-attack -d ${scorepath_dev}dev-real -f ${scorepath}eval-attack -e ${scorepath}eval-real -o plots_${p}_${f}_20"
plotnames+=("${ed}_${f}")
command="-n 10 -m 60 -t ${scorepath_dev}-attack -d ${scorepath_dev}-real -f ${scorepath_eval}-attack -e
${scorepath_eval}-real -o plots_${ed}_${f}_20"
echo $command --pretty-title "${prettytitles[indxfeat]}${prettypostfixes[indxfixes]}"
bin/pad_process_scores.py $command --pretty-title "${prettytitles[indxfeat]}${prettypostfixes[indxfixes]}"
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment