Commit b058fd4a authored by Amir MOHAMMADI's avatar Amir MOHAMMADI
Browse files

Initial database

parents
*~
*.swp
*.pyc
bin
eggs
parts
.installed.cfg
.mr.developer.cfg
*.egg-info
src
develop-eggs
sphinx
dist
*.sql3
20170512-110547/
test:
variables:
CONDA_ENVS_PATH: "conda-env"
CONDA_BLD_PATH: "conda-env"
script:
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda info -a
# https://www.idiap.ch does not work on CI machines but works elsewhere
- sed -i "s|https://www.idiap.ch|http://www.idiap.ch|g" environment.yml
- conda env create -vvv --file environment.yml
- source activate bob.paper.cvpr2018_facevuln
# this is a private link (works on the CI machine only)
- curl -o 20170512-110547.zip http://www.idiap.ch/private/wheels/gitlab/facenet_model2_20170512-110547.zip
- unzip 20170512-110547.zip
- buildout
- bin/bob_dbmanage.py all download --missing
- bin/nosetests -sv .
cache:
key: "$CI_BUILD_NAME"
paths:
- conda-env/.pkgs/*.tar.bz2
- conda-env/.pkgs/urls.txt
- conda-env/src_cache
image: continuumio/miniconda
tags:
- docker
This diff is collapsed.
include README.rst buildout.cfg COPYING version.txt requirements.txt
recursive-include doc *.py *.rst *.ico *.png
.. vim: set fileencoding=utf-8 :
.. Mon Sep 11 10:23:39 CEST 2017
=======================================================================
A Study of the Robustness of Face Recognition to Presentation Attacks
=======================================================================
This package is part of the signal-processing and machine learning toolbox
Bob_. It contains the source code to reproduce the following paper::
@article{Mohammadi_Deeply_Vulnerable_2017,
title = {Deeply Vulnerable -- A Study of the Robustness of Face Recognition to Presentation Attacks},
author = {Mohammadi, Amir and Bhattacharjee, Sushil and Marcel, S{\'{e}}bastien},
journal = {IET biometrics},
year = {2017},
institution = {Idiap},
}
Installation
------------
The installation instructions are based on conda_ and works on **Linux systems
only**. `Install conda`_ before continuing.
Once you have installed conda_, download the source code of this paper and
unpack it. Then, you can create a conda environment with the following
command::
$ cd bob.paper.cvpr2018_facevuln
$ conda env create -f environment.yml
$ source activate bob.paper.cvpr2018_facevuln # activate the environment
$ buildout
$ bin/bob_dbmanage.py all download --missing
$ bin/nosetests -sv . # test the installation
This will install all the required software to reproduce this paper.
Downloading the datasets
------------------------
Four datasets are used in this study which are publicly available.
To download the datasets please refer to their websites:
* `Mobio`_
* `Replay Attack`_
* `Replay Mobile`_
* `MSU MFSD`_
Downloading the face recognition models
---------------------------------------
Pre-trained face recognition (FR) models can be downloaded from their
respective website as well.
* `VGG Face`_ (`VGG Face model`_)
* `LightCNN`_ (`LightCNN model`_)
* `FaceNet`_ (`FaceNet model`_)
The code will automatically download the VGG Face and LightCNN models and will
place them in the right place. So you may skip downloading those manually.
Please unzip the `FaceNet model`_ in the source directory with the name
``20170512-110547``.
ROC SDK experiments
-------------------
`ROC SDK`_ version 1.9 is used in the paper.
The `ROC SDK`_ is a commercial product and a license is needed to use
it. You can skip the `ROC SDK`_ experiments if you do not have a license.
If you do, you will also need the Bob wrapper packages for using the ROC SDK.
These packages are available upon request.
Configuring the experiments
---------------------------
Now that you have downloaded the four databases. You need to set the paths to
those in the configuration files. Bob_ supports a configuration file
(``~/.bob_bio_databases.txt``) in your home directory to specify where the
databases are located. Please specify the paths for the database like below::
$ cat ~/.bob_bio_databases.txt
[YOUR_MOBIO_IMAGE_DIRECTORY] = /databases/mobio/IMAGES_PNG/
[YOUR_MOBIO_ANNOTATION_DIRECTORY] = /databases/mobio/IMAGE_ANNOTATIONS/
[YOUR_MSU_MFSD_MOD_DIRECTORY] = /databases/MSU-MFSD/scene01/
[YOUR_REPLAY_ATTACK_DIRECTORY] = /databases/replay/protocols/replayattack-database/
[YOUR_REPLAY_MOBILE_DIRECTORY] = /databases/replay-mobile/database/
Running the experiments
-----------------------
Follow the commands below to run the computational part of the experiments with
`bob.bio.base`_.
VGG::
$ bin/verify.py config_base.py config_cnn_eyes.py config_mobio.py
$ bin/verify.py config_base.py config_cnn.py config_replay_licit.py
$ bin/verify.py config_base.py config_cnn.py config_replay_spoof.py
$ bin/verify.py config_base.py config_cnn.py config_replaymobile_licit.py
$ bin/verify.py config_base.py config_cnn.py config_replaymobile_spoof.py
$ bin/verify.py config_base.py config_cnn_eyes.py config_msumfsd_licit.py
$ bin/verify.py config_base.py config_cnn_eyes.py config_msumfsd_spoof.py
LightCNN::
$ bin/verify.py config_base.py config_lightcnn_eyes.py config_mobio.py
$ bin/verify.py config_base.py config_lightcnn.py config_replay_licit.py
$ bin/verify.py config_base.py config_lightcnn.py config_replay_spoof.py
$ bin/verify.py config_base.py config_lightcnn.py config_replaymobile_licit.py
$ bin/verify.py config_base.py config_lightcnn.py config_replaymobile_spoof.py
$ bin/verify.py config_base.py config_lightcnn_eyes.py config_msumfsd_licit.py
$ bin/verify.py config_base.py config_lightcnn_eyes.py config_msumfsd_spoof.py
FaceNet::
$ bin/verify.py config_base.py config_facenet_eyes.py config_mobio.py
$ bin/verify.py config_base.py config_facenet.py config_replay_licit.py
$ bin/verify.py config_base.py config_facenet.py config_replay_spoof.py
$ bin/verify.py config_base.py config_facenet.py config_replaymobile_licit.py
$ bin/verify.py config_base.py config_facenet.py config_replaymobile_spoof.py
$ bin/verify.py config_base.py config_facenet_eyes.py config_msumfsd_licit.py
$ bin/verify.py config_base.py config_facenet_eyes.py config_msumfsd_spoof.py
ISV::
$ bin/verify_isv.py config_base.py config_isv_eyes.py config_mobio.py -s isv-male
$ bin/verify_isv.py config_base.py config_isv_topleft.py config_replay_licit.py
$ bin/verify_isv.py config_base.py config_isv_topleft.py config_replay_spoof.py
$ bin/verify_isv.py config_base.py config_isv_topleft.py config_replaymobile_licit.py
$ bin/verify_isv.py config_base.py config_isv_topleft.py config_replaymobile_spoof.py
$ bin/verify_isv.py config_base.py config_isv_eyes.py config_msumfsd_licit.py
$ bin/verify_isv.py config_base.py config_isv_eyes.py config_msumfsd_spoof.py
RANKONE::
$ bin/verify.py config_base.py config_rankone.py config_mobio.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_replay_licit.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_replay_spoof.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_replaymobile_licit.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_replaymobile_spoof.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_msumfsd_licit.py
$ bin/verify.py config_base.py config_rankone_loaded.py config_msumfsd_spoof.py
You can look at ``bin/verify.py --help`` to find options for parallelization of
the runs. By default results are saved in the ``./results`` directory::
$ export RESULTS_DIR=./results
The score files (output of ``verify.py`` commands) are also provided if you do
not want to run all the experiments again. You can extract them from
``./results/scores.tar.xz``. The score files can be used to generate the tables
and figures.
Generating the figures and tables
---------------------------------
The ROC and EPC curves on the Mobio dataset::
$ bin/evaluate.py -vvv \
-M 1e-5 -L 1e-3 -c EER \
-R mobio_nonorm_roc.pdf \
-E mobio_nonorm_epc.pdf \
-l VGG-Face LightCNN FaceNet ROC-SDK ISV \
-T "" "" \
-d \
$RESULTS_DIR/mobio/vgg-cosine/male/nonorm/scores-dev \
$RESULTS_DIR/mobio/lightcnn-cosine/male/nonorm/scores-dev \
$RESULTS_DIR/mobio/facenet-cosine/male/nonorm/scores-dev \
$RESULTS_DIR/mobio/rankone/male/nonorm/scores-dev \
$RESULTS_DIR/mobio/isv-male/male/nonorm/scores-dev \
-e \
$RESULTS_DIR/mobio/vgg-cosine/male/nonorm/scores-eval \
$RESULTS_DIR/mobio/lightcnn-cosine/male/nonorm/scores-eval \
$RESULTS_DIR/mobio/facenet-cosine/male/nonorm/scores-eval \
$RESULTS_DIR/mobio/rankone/male/nonorm/scores-eval \
$RESULTS_DIR/mobio/isv-male/male/nonorm/scores-eval
Score distributions of ROC-SDK and VGG-Face on Mobio::
$ bin/threshold_evolution.py $RESULTS_DIR/mobio/rankone/male/nonorm/scores-{dev,eval} -o mobio_nonorm_rankone_hist.pdf
$ bin/threshold_evolution.py $RESULTS_DIR/mobio/vgg-cosine/male/nonorm/scores-{dev,eval} -o mobio_nonorm_vgg-cosine_hist.pdf
Vulnerability analysis tables::
$ # concatenate scores of all three PA datasets:
$ for licit in "licit" "spoof";
do
for dev in "dev" "eval";
do
for alg in "facenet-cosine" "isv" "lightcnn-cosine" "rankone" "vgg-cosine";
do
cat $RESULTS_DIR/replay/$alg/grandtest-${licit}/nonorm/scores-${dev} \
$RESULTS_DIR/msu-mfsd-mod/$alg/grandtest-${licit}/nonorm/scores-${dev} \
$RESULTS_DIR/replay-mobile/$alg/grandtest-${licit}/nonorm/scores-${dev} \
> $RESULTS_DIR/three-$alg-${licit}-${dev};
done
done
done
$ # print the table values:
$ echo "replay attack"
$ for alg in "vgg-cosine" "lightcnn-cosine" "facenet-cosine" "rankone" "isv" ; do echo $alg && bin/vulnerability.py $RESULTS_DIR/replay/$alg/grandtest-{licit,spoof}/nonorm/scores-{dev,eval}; done
$ echo "msu mfsd"
$ for alg in "vgg-cosine" "lightcnn-cosine" "facenet-cosine" "rankone" "isv" ; do echo $alg && bin/vulnerability.py $RESULTS_DIR/msu-mfsd-mod/$alg/grandtest-{licit,spoof}/nonorm/scores-{dev,eval}; done
$ echo "replay mobile"
$ for alg in "vgg-cosine" "lightcnn-cosine" "facenet-cosine" "rankone" "isv" ; do echo $alg && bin/vulnerability.py $RESULTS_DIR/replay-mobile/$alg/grandtest-{licit,spoof}/nonorm/scores-{dev,eval}; done
$ echo "all datasets"
$ for alg in "vgg-cosine" "lightcnn-cosine" "facenet-cosine" "rankone" "isv" ; do echo $alg && bin/vulnerability.py $RESULTS_DIR/three-$alg-{licit,spoof}-{dev,eval}; done
Score distribution of all three PA datasets combined::
$ bin/plot_on_demand_better.py -v $RESULTS_DIR/three-{vgg-cosine,lightcnn-cosine,facenet-cosine,rankone,isv}-{licit,spoof}-{dev,eval} -t VGG-Face,LightCNN,FaceNet,ROC-SDK,ISV -o allthree_dist.pdf
Contact
-------
For questions or reporting issues to this software package, contact our
development `mailing list`_.
.. Place your references here:
.. _bob: https://www.idiap.ch/software/bob
.. _mailing list: https://www.idiap.ch/software/bob/discuss
.. _conda: https://conda.io
.. _install conda: https://conda.io/docs/install/quick.html#linux-miniconda-install
.. _bob.bio.base: https://pypi.python.org/pypi/bob.bio.base
.. _paper: http://publications.idiap.ch/index.php/publications/show/$$$$
.. _mobio: https://www.idiap.ch/dataset/mobio
.. _replay attack: http://www.idiap.ch/dataset/replayattack
.. _replay mobile: http://www.idiap.ch/dataset/replay-mobile
.. _msu mfsd: https://www.cse.msu.edu/rgroups/biometrics/Publications/Databases/MSUMobileFaceSpoofing/index.htm
.. _vgg face: http://www.robots.ox.ac.uk/~vgg/software/vgg_face
.. _vgg face model: http://www.robots.ox.ac.uk/~vgg/software/vgg_face
.. _lightcnn: https://github.com/AlfredXiangWu/face_verification_experiment
.. _lightcnn model: https://github.com/AlfredXiangWu/face_verification_experiment/archive/master.zip
.. _facenet: https://github.com/davidsandberg/facenet
.. _facenet model: https://drive.google.com/uc?export=download&confirm=YCEV&id=0B5MzpY9kBtDVZ2RpVDYwWmxoSUk
.. _roc sdk: https://www.rankone.io
# see https://docs.python.org/3/library/pkgutil.html
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
# see https://docs.python.org/3/library/pkgutil.html
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
from . import script
from .utils import *
def get_config():
"""Returns a string containing the configuration information.
"""
import bob.extension
return bob.extension.get_config(__name__)
# gets sphinx autodoc done right - don't remove it
__all__ = [_ for _ in dir() if not _.startswith('_')]
from .query import Database
def __appropriate__(*args):
"""Says object was actually declared here, and not in the import module.
Fixing sphinx warnings of not being able to find classes, when path is
shortened. Parameters:
*args: An iterable of objects to modify
Resolves `Sphinx referencing issues
<https://github.com/sphinx-doc/sphinx/issues/3048>`
"""
for obj in args:
obj.__module__ = __name__
__appropriate__(
Database,
)
# gets sphinx autodoc done right - don't remove it
__all__ = [_ for _ in dir() if not _.startswith('_')]
#!/usr/bin/env python
"""Takes a list bona-fide files and generates their zei list.
Usage:
%(prog)s <infile> <outfile>
%(prog)s -h | --help
Options:
-h --help Show this screen.
"""
from collections import defaultdict
import six
def convert(infile, outfile):
bonafides = defaultdict(list)
with open(infile, 'rt') as f:
for line in f:
path, client_id = line.split()
bonafides[client_id].append(path)
zeis = defaultdict(list)
all_clients = list(bonafides.keys())
for bf_client_id, paths in bonafides.items():
for client_id in all_clients:
if client_id == bf_client_id:
continue
zeis[client_id] += six.moves.zip_longest(
paths, [bf_client_id], fillvalue=bf_client_id)
with open(outfile, 'wt') as f:
for client_id, paths in zeis.items():
for path, bf_client_id in paths:
f.write('{0} {1} {1} {2}\n'.format(
path, client_id, bf_client_id))
def main():
import sys
import os
import docopt
completions = dict(
prog=os.path.basename(sys.argv[0]),
)
args = docopt.docopt(
__doc__ % completions,
)
convert(args['<infile>'], args['<outfile>'])
if __name__ == '__main__':
main()
from bob.paper.cvpr2018_facevuln.database import Database
ORIGINAL_DIRECTORY = '[SILICONECVPR_3DMask_Data_DIRECTORY]'
database_licit = Database(original_directory=ORIGINAL_DIRECTORY, protocol='licit')
database_spoof = Database(original_directory=ORIGINAL_DIRECTORY, protocol='spoof')
Bonafide/A/photos/no_coop/DSCN3121.JPG A
Bonafide/A/photos/with_coop/DSCN3115.JPG A
Bonafide/B/photos/no_coop/DSCN3100.JPG B
Bonafide/B/photos/with_coop/DSCN3095.JPG B
Bonafide/C/photos/no_coop/DSCN3188.JPG C
Bonafide/C/photos/with_coop/DSCN3167.JPG C
Bonafide/C/photos/with_coop/DSCN3192.JPG C
Bonafide/D/photos/with_coop/DSCN3105.JPG D
Bonafide/E/photos/no_coop/DSCN3148.JPG E
Bonafide/E/photos/no_coop/DSCN3149.JPG E
Bonafide/E/photos/no_coop/DSCN3150.JPG E
Bonafide/E/photos/no_coop/DSCN3151.JPG E
Bonafide/E/photos/no_coop/DSCN3152.JPG E
Bonafide/E/photos/with_coop/DSCN3142.JPG E
Bonafide/F/photos/no_coop/DSCN3131.JPG F
Bonafide/F/photos/no_coop/DSCN3132.JPG F
Bonafide/F/photos/with_coop/DSCN3126.JPG F
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 E
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 F
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i1_062.h5 A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i2_063.h5 A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i3_064.h5 A
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_gen_i0_006.h5 B
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_gen_i3_002.h5 B
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_Gen_i2_001.h5 B
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i0_001.h5 D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i1_003.h5 D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i2_004.h5 D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i3_005.h5 D
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i0_005.h5 B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i1_006.h5 B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i2_007.h5 B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i3_008.h5 B
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i0_001.h5 D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i1_002.h5 D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i2_003.h5 D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i3_004.h5 D
Bonafide/A/photos/with_coop/DSCN3115.JPG A A
Bonafide/B/photos/with_coop/DSCN3095.JPG B B
Bonafide/C/photos/with_coop/DSCN3167.JPG C C
Bonafide/C/photos/with_coop/DSCN3192.JPG C C
Bonafide/D/photos/with_coop/DSCN3105.JPG D D
Bonafide/E/photos/with_coop/DSCN3142.JPG E E
Bonafide/F/photos/with_coop/DSCN3126.JPG F F
Bonafide/A/photos/no_coop/DSCN3121.JPG A A A
Bonafide/B/photos/no_coop/DSCN3100.JPG B B B
Bonafide/C/photos/no_coop/DSCN3188.JPG C C C
Bonafide/E/photos/no_coop/DSCN3148.JPG E E E
Bonafide/E/photos/no_coop/DSCN3149.JPG E E E
Bonafide/E/photos/no_coop/DSCN3150.JPG E E E
Bonafide/E/photos/no_coop/DSCN3151.JPG E E E
Bonafide/E/photos/no_coop/DSCN3152.JPG E E E
Bonafide/F/photos/no_coop/DSCN3131.JPG F F F
Bonafide/F/photos/no_coop/DSCN3132.JPG F F F
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 E E E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 E E E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 E E E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 E E E
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 F F F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 F F F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 F F F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 F F F
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 A A A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i1_062.h5 A A A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i2_063.h5 A A A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i3_064.h5 A A A
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_gen_i0_006.h5 B B B
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_gen_i3_002.h5 B B B
Mask_Batl_Data4_BonaFide/captured/no_glasses/B_Gen_i2_001.h5 B B B
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i0_001.h5 D D D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i1_003.h5 D D D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i2_004.h5 D D D
Mask_Batl_Data4_BonaFide/captured/no_glasses/D_gen_i3_005.h5 D D D
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i0_005.h5 B B B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i1_006.h5 B B B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i2_007.h5 B B B
Mask_Batl_Data4_BonaFide/captured/withglasses/B_genglasses_i3_008.h5 B B B
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i0_001.h5 D D D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i1_002.h5 D D D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i2_003.h5 D D D
Mask_Batl_Data4_BonaFide/captured/withglasses/D_genglasses_i3_004.h5 D D D
Bonafide/A/photos/no_coop/DSCN3121.JPG B B A
Bonafide/A/photos/no_coop/DSCN3121.JPG C C A
Bonafide/A/photos/no_coop/DSCN3121.JPG D D A
Bonafide/A/photos/no_coop/DSCN3121.JPG E E A
Bonafide/A/photos/no_coop/DSCN3121.JPG F F A
Bonafide/A/photos/with_coop/DSCN3115.JPG B B A
Bonafide/A/photos/with_coop/DSCN3115.JPG C C A
Bonafide/A/photos/with_coop/DSCN3115.JPG D D A
Bonafide/A/photos/with_coop/DSCN3115.JPG E E A
Bonafide/A/photos/with_coop/DSCN3115.JPG F F A
Bonafide/B/photos/no_coop/DSCN3100.JPG A A B
Bonafide/B/photos/no_coop/DSCN3100.JPG C C B
Bonafide/B/photos/no_coop/DSCN3100.JPG D D B
Bonafide/B/photos/no_coop/DSCN3100.JPG E E B
Bonafide/B/photos/no_coop/DSCN3100.JPG F F B
Bonafide/B/photos/with_coop/DSCN3095.JPG A A B
Bonafide/B/photos/with_coop/DSCN3095.JPG C C B
Bonafide/B/photos/with_coop/DSCN3095.JPG D D B
Bonafide/B/photos/with_coop/DSCN3095.JPG E E B
Bonafide/B/photos/with_coop/DSCN3095.JPG F F B
Bonafide/C/photos/no_coop/DSCN3188.JPG A A C
Bonafide/C/photos/no_coop/DSCN3188.JPG B B C
Bonafide/C/photos/no_coop/DSCN3188.JPG D D C
Bonafide/C/photos/no_coop/DSCN3188.JPG E E C
Bonafide/C/photos/no_coop/DSCN3188.JPG F F C
Bonafide/C/photos/with_coop/DSCN3167.JPG A A C
Bonafide/C/photos/with_coop/DSCN3167.JPG B B C
Bonafide/C/photos/with_coop/DSCN3167.JPG D D C
Bonafide/C/photos/with_coop/DSCN3167.JPG E E C
Bonafide/C/photos/with_coop/DSCN3167.JPG F F C
Bonafide/C/photos/with_coop/DSCN3192.JPG A A C
Bonafide/C/photos/with_coop/DSCN3192.JPG B B C
Bonafide/C/photos/with_coop/DSCN3192.JPG D D C
Bonafide/C/photos/with_coop/DSCN3192.JPG E E C
Bonafide/C/photos/with_coop/DSCN3192.JPG F F C
Bonafide/D/photos/with_coop/DSCN3105.JPG A A D
Bonafide/D/photos/with_coop/DSCN3105.JPG B B D
Bonafide/D/photos/with_coop/DSCN3105.JPG C C D
Bonafide/D/photos/with_coop/DSCN3105.JPG E E D
Bonafide/D/photos/with_coop/DSCN3105.JPG F F D
Bonafide/E/photos/no_coop/DSCN3148.JPG A A E
Bonafide/E/photos/no_coop/DSCN3148.JPG B B E
Bonafide/E/photos/no_coop/DSCN3148.JPG C C E
Bonafide/E/photos/no_coop/DSCN3148.JPG D D E
Bonafide/E/photos/no_coop/DSCN3148.JPG F F E
Bonafide/E/photos/no_coop/DSCN3149.JPG A A E
Bonafide/E/photos/no_coop/DSCN3149.JPG B B E
Bonafide/E/photos/no_coop/DSCN3149.JPG C C E
Bonafide/E/photos/no_coop/DSCN3149.JPG D D E
Bonafide/E/photos/no_coop/DSCN3149.JPG F F E
Bonafide/E/photos/no_coop/DSCN3150.JPG A A E
Bonafide/E/photos/no_coop/DSCN3150.JPG B B E
Bonafide/E/photos/no_coop/DSCN3150.JPG C C E
Bonafide/E/photos/no_coop/DSCN3150.JPG D D E
Bonafide/E/photos/no_coop/DSCN3150.JPG F F E
Bonafide/E/photos/no_coop/DSCN3151.JPG A A E
Bonafide/E/photos/no_coop/DSCN3151.JPG B B E
Bonafide/E/photos/no_coop/DSCN3151.JPG C C E
Bonafide/E/photos/no_coop/DSCN3151.JPG D D E
Bonafide/E/photos/no_coop/DSCN3151.JPG F F E
Bonafide/E/photos/no_coop/DSCN3152.JPG A A E
Bonafide/E/photos/no_coop/DSCN3152.JPG B B E
Bonafide/E/photos/no_coop/DSCN3152.JPG C C E
Bonafide/E/photos/no_coop/DSCN3152.JPG D D E
Bonafide/E/photos/no_coop/DSCN3152.JPG F F E
Bonafide/E/photos/with_coop/DSCN3142.JPG A A E
Bonafide/E/photos/with_coop/DSCN3142.JPG B B E
Bonafide/E/photos/with_coop/DSCN3142.JPG C C E
Bonafide/E/photos/with_coop/DSCN3142.JPG D D E
Bonafide/E/photos/with_coop/DSCN3142.JPG F F E
Bonafide/F/photos/no_coop/DSCN3131.JPG A A F
Bonafide/F/photos/no_coop/DSCN3131.JPG B B F
Bonafide/F/photos/no_coop/DSCN3131.JPG C C F
Bonafide/F/photos/no_coop/DSCN3131.JPG D D F
Bonafide/F/photos/no_coop/DSCN3131.JPG E E F
Bonafide/F/photos/no_coop/DSCN3132.JPG A A F
Bonafide/F/photos/no_coop/DSCN3132.JPG B B F
Bonafide/F/photos/no_coop/DSCN3132.JPG C C F
Bonafide/F/photos/no_coop/DSCN3132.JPG D D F
Bonafide/F/photos/no_coop/DSCN3132.JPG E E F
Bonafide/F/photos/with_coop/DSCN3126.JPG A A F
Bonafide/F/photos/with_coop/DSCN3126.JPG B B F
Bonafide/F/photos/with_coop/DSCN3126.JPG C C F
Bonafide/F/photos/with_coop/DSCN3126.JPG D D F
Bonafide/F/photos/with_coop/DSCN3126.JPG E E F
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 A A E
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 B B E
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 C C E
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 D D E
Mask_Batl_Data/bonafide/E_gen_i0_003.h5 F F E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 A A E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 B B E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 C C E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 D D E
Mask_Batl_Data/bonafide/E_gen_i1_004.h5 F F E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 A A E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 B B E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 C C E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 D D E
Mask_Batl_Data/bonafide/E_gen_i2_005.h5 F F E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 A A E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 B B E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 C C E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 D D E
Mask_Batl_Data/bonafide/E_gen_i3_006.h5 F F E
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 A A F
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 B B F
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 C C F
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 D D F
Mask_Batl_Data/bonafide/F_gen_i0_007.h5 E E F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 A A F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 B B F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 C C F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 D D F
Mask_Batl_Data/bonafide/F_gen_i1_008.h5 E E F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 A A F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 B B F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 C C F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 D D F
Mask_Batl_Data/bonafide/F_gen_i2_009.h5 E E F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 A A F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 B B F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 C C F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 D D F
Mask_Batl_Data/bonafide/F_gen_i3_010.h5 E E F
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 B B A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 C C A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 D D A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 E E A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i0_061.h5 F F A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i1_062.h5 B B A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i1_062.h5 C C A
Mask_Batl_Data4_BonaFide/captured/no_glasses/A_gen_i1_062.h5 D D A