Commit 23010aa2 authored by Tiago de Freitas Pereira's avatar Tiago de Freitas Pereira

Merge branch 'multipie-doc' into 'master'

Multipie Leaderboard

See merge request !95
parents 20cce939 7674e53b
Pipeline #47412 passed with stages
in 45 minutes and 37 seconds
.. vim: set fileencoding=utf-8 :
.. _bob.bio.face.learderboard.multipie:
.. _bob.bio.face.leaderboard.multipie:
================
Multipie Dataset
================
.. todo::
Benchmarks on Multipie Database
Probably for Manuel's students
\ No newline at end of file
Probably for Manuel's students
Database
========
The `CMU Multi-PIE face database <http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html>`_ contains more than 750,000 images
of 337 people recorded in up to four sessions over the span of five months. Subjects were imaged under 15 view points and 19 illumination
conditions while displaying a range of facial expressions. In addition, high resolution frontal images were acquired as well.
In total, the database contains more than 305 GB of face data.
Content
*******
The data has been recorded over 4 sessions. For each session, the subjects were asked to display a few
different expressions. For each of those expressions, a complete set of 30 pictures is captured that includes
15 different view points times 20 different illumination conditions (18 with various flashes, plus 2 pictures with no flash at all).
Available expressions
---------------------
* Session 1 : *neutral*, *smile*
* Session 2 : *neutral*, *surprise*, *squint*
* Session 3 : *neutral*, *smile*, *disgust*
* Session 4 : *neutral*, *neutral*, *scream*.
Camera and flash positioning
----------------------------
The different view points are obtained by a set of 13 cameras located at head height, spaced at 15° intervals,
from the -90° to the 90° angle, plus 2 additional cameras located above the subject to simulate a typical
surveillance view. A flash coincides with each camera, and 3 additional flashes are positioned above the subject, for a total
of 18 different possible flashes.
The following picture showcase the positioning of the cameras (in yellow) and of the flashes (in white).
.. image:: img/multipie/multipie_setup.jpg
:width: 620px
:align: center
:height: 200px
:alt: Multipie setup
File paths
----------
The data directory structure and filenames adopt the following structure:
.. code-block:: shell
session<XX>/multiview/<subject_id>/<recording_id>/<camera_id>/<subject_id>_<session_id>_<recording_id>_<camera_id>_<shot_id>.png
For example, the file
.. code-block:: shell
session02/multiview/001/02/05_1/001_02_02_051_07.png
corresponds to
* Subject 001
* Session 2
* Second recording -> Expression is *surprise*
* Camera 05_1 -> Frontal view
* Shot 07 -> Illumination through the frontal flash
Protocols
*********
Expression protocol
-------------------
**Protocol E**
* Only frontal view (camera 05_1); only no-flash (shot 0)
* Enrolled : 1x neutral expression (session 1; recording 1)
* Probes : 4x neutral expression + other expressions (session 2, 3, 4; all recordings)
Pose protocol
-------------
**Protocol P**
* Only neutral expression (recording 1 from each session, + recording 2 from session 4); only no-flash (shot 0)
* Enrolled : 1x frontal view (session 1; camera 05_1)
* Probes : all views from cameras at head height (i.e excluding 08_1 and 19_1), including camera 05_1 from session 2,3,4.
Illumination protocols
----------------------
N.B : shot 19 is never used in those protocols as it is redundant with shot 0 (both are no-flash).
**Protocol M**
* Only frontal view (camera 05_1); only neutral expression (recording 1 from each session, + recording 2 from session 4)
* Enrolled : no-flash (session 1; shot 0)
* Probes : no-flash (session 2, 3, 4; shot 0)
**Protocol U**
* Only frontal view (camera 05_1); only neutral expression (recording 1 from each session, + recording 2 from session 4)
* Enrolled : no-flash (session 1; shot 0)
* Probes : all shots from session 2, 3, 4, including shot 0.
**Protocol G**
* Only frontal view (camera 05_1); only neutral expression (recording 1 from each session, + recording 2 from session 4)
* Enrolled : all shots (session 1; all shots)
* Probes : all shots from session 2, 3, 4.
Benchmarks
==========
Run the baselines
*****************
You can run the Multipie baselines command with a simple command such as:
.. code-block:: bash
bob bio pipeline vanilla-biometrics multipie gabor_graph -m -l sge
Note that the default protocol implemented in the resource is the U protocol.
The pose protocol is also available using
.. code-block:: bash
bob bio pipeline vanilla-biometrics multipie_pose gabor_graph -m -l sge
For the other protocols, one has to define its own configuration file (e.g.: *multipie_M.py*) as follows:
.. code-block:: python
from bob.bio.face.database import MultipieDatabase
database = MultipieDatabase(protocol="M")
then point to it when calling the pipeline execution:
.. code-block:: bash
bob bio pipeline vanilla-biometrics multipie_M.py gabor_graph -m -l sge
Leaderboard
***********
Protocol M
----------
.. csv-table:: Protocol M
:file: table/multipie/multipie_M.csv
:header-rows: 1
.. image:: img/multipie/multipie_M_DET_dev.png
:align: center
:alt: Multipie M - DET dev
.. image:: img/multipie/multipie_M_DET_eval.png
:align: center
:alt: Multipie M - DET eval
Protocol U
----------
.. csv-table:: Protocol U
:file: table/multipie/multipie_U.csv
:header-rows: 1
.. image:: img/multipie/multipie_U_DET_dev.png
:align: center
:alt: Multipie U - DET dev
.. image:: img/multipie/multipie_U_DET_eval.png
:align: center
:alt: Multipie U - DET eval
Protocol E
----------
.. csv-table:: Protocol E
:file: table/multipie/multipie_E.csv
:header-rows: 1
.. image:: img/multipie/multipie_E_DET_dev.png
:align: center
:alt: Multipie E - DET dev
.. image:: img/multipie/multipie_E_DET_eval.png
:align: center
:alt: Multipie E - DET eval
Protocol P
----------
.. csv-table:: Protocol P
:file: table/multipie/multipie_P.csv
:header-rows: 1
.. image:: img/multipie/multipie_P_DET_dev.png
:align: center
:alt: Multipie P - DET dev
.. image:: img/multipie/multipie_P_DET_eval.png
:align: center
:alt: Multipie P - DET eval
For the pose protocol specifically, we can perform a more detailed study
to assess angle-wise performance of the various FR systems.
Hereafter is an example code to run this type of analysis, as well
as the results. This code is also available as a Jupytext-compatible
.py file under `./script/multipie/pose_analysis.py`, that can be loaded
as a Jupyter notebook.
.. raw:: html
:file: script/multipie/pose_analysis.html
This diff is collapsed.
# ---
# jupyter:
# jupytext:
# formats: ipynb,py:light
# text_representation:
# extension: .py
# format_name: light
# format_version: '1.5'
# jupytext_version: 1.7.1
# kernelspec:
# display_name: bob_pipelines
# language: python
# name: bob_pipelines
# ---
import os
import pandas as pd
import bob.measure
import numpy as np
import matplotlib as mpl
mpl.rcParams.update({'font.size': 14})
import matplotlib.pyplot as plt
# %matplotlib inline
# ### Select baselines
baselines = {
'Gabor graph': 'gabor_graph',
'LGBPHS': 'lgbphs',
'FaceNet (D. Sandberg)': 'facenet-sanderberg',
'Casia-Webface - Inception Resnet v1': 'inception-resnetv1-casiawebface',
'Casia-Webface - Inception Resnet v2': 'inception-resnetv2-casiawebface',
'MSCeleb - Inception Resnet v1': 'inception-resnetv1-msceleb',
'MSCeleb - Inception Resnet v2': 'inception-resnetv2-msceleb'
}
# ### Define some utilitary functions
# +
results_dir = './results/'
# Function to load the scores in CSV format
def load_scores(baseline, protocol, group):
scores = pd.read_csv(os.path.join(results_dir,
'multipie_{}'.format(protocol),
baseline,
'scores-{}.csv'.format(group)))
return scores
# Function to separate genuines from impostors
def split(df):
impostors = df[df['probe_reference_id'] != df['bio_ref_reference_id']]
genuines = df[df['probe_reference_id'] == df['bio_ref_reference_id']]
return impostors, genuines
# -
# ### Establish Camera to Angle conversion
cameras = ['11_0', '12_0','09_0','08_0','13_0','14_0','05_1','05_0', '04_1','19_0','20_0','01_0','24_0']
angles = np.linspace(-90, 90, len(cameras))
camera_to_angle = dict(zip(cameras, angles))
# ### Run pose analysis and plot results
# +
# Figure for Dev plot
fig1 = plt.figure(figsize=(8, 6))
# Figure for Eval plot
fig2 = plt.figure(figsize=(8, 6))
for name, baseline in baselines.items():
# Load the score files and fill in the angle associated to each camera
dev_scores = load_scores(baseline, 'P', 'dev')
eval_scores = load_scores(baseline, 'P', 'eval')
dev_scores['angle'] = dev_scores['probe_camera'].map(camera_to_angle)
eval_scores['angle'] = eval_scores['probe_camera'].map(camera_to_angle)
angles = []
dev_hters = []
eval_hters = []
# Run the analysis per view angle
for (angle, dev_df), (_, eval_df) in zip(dev_scores.groupby('angle'), eval_scores.groupby('angle')):
# Separate impostors from genuines
dev_impostors, dev_genuines = split(dev_df)
eval_impostors, eval_genuines = split(eval_df)
# Compute the min. HTER threshold on the Dev set
threshold = bob.measure.min_hter_threshold(dev_impostors['score'], dev_genuines['score'])
# Compute the HTER for the Dev and Eval set at this particular threshold
dev_far, dev_frr = bob.measure.farfrr(dev_impostors['score'], dev_genuines['score'], threshold)
eval_far, eval_frr = bob.measure.farfrr(eval_impostors['score'], eval_genuines['score'], threshold)
angles.append(angle)
dev_hters.append(1/2 * (dev_far + dev_frr))
eval_hters.append(1/2 * (eval_far + eval_frr))
# Update plots
plt.figure(1)
plt.plot(angles, dev_hters, label=name, marker='x')
plt.figure(2)
plt.plot(angles, eval_hters, label=name, marker='x')
# Plot finalization
plt.figure(1)
plt.title('Dev. min. HTER')
plt.xlabel('Angle')
plt.ylabel('Min HTER')
plt.legend()
plt.grid()
plt.figure(2)
plt.title('Eval. HTER @Dev min. HTER')
plt.xlabel('Angle')
plt.ylabel('HTER')
plt.legend()
plt.grid()
Baseline,EER (dev),HTER (eval)
Gabor graph,14.24%,15.56%
LGBPHS,13.36%,13.78%
FaceNet (D. Sandberg),5.21%,6.64%
Casia-Webface - Inception Resnet v1,12.89%,13.67%
Casia-Webface - Inception Resnet v2,2.11%,3.07%
MSCeleb - Inception Resnet v1,6.27%,6.60%
MSCeleb - Inception Resnet v2,3.82%,4.46%
Baseline,EER (dev),HTER (eval)
Gabor graph,1.26%,3.68%
LGBPHS,1.17%,3.06%
FaceNet (D. Sandberg),1.56%,2.74%
Casia-Webface - Inception Resnet v1,3.83%,5.98%
Casia-Webface - Inception Resnet v2,0.76%,0.74%
MSCeleb - Inception Resnet v1,1.56%,2.19%
MSCeleb - Inception Resnet v2,3.09%,7.09%
Baseline,EER (dev),HTER (eval)
Gabor graph,44.83%,45.66%
LGBPHS,48.23%,48.36%
FaceNet (D. Sandberg),14.46%,13.93%
Casia-Webface - Inception Resnet v1,37.23%,37.11%
Casia-Webface - Inception Resnet v2,13.04%,13.12%
MSCeleb - Inception Resnet v1,13.58%,14.76%
MSCeleb - Inception Resnet v2,5.17%,5.22%
Baseline,EER (dev),HTER (eval)
Gabor graph,6.25%,7.44%
LGBPHS,6.02%,6.85%
FaceNet (D. Sandberg),3.31%,3.56%
Casia-Webface - Inception Resnet v1,9.50%,9.43%
Casia-Webface - Inception Resnet v2,0.80%,1.46%
MSCeleb - Inception Resnet v1,3.97%,3.10%
MSCeleb - Inception Resnet v2,3.82%,6.54%
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment