updated notebooks

notebooks/model.07_asv_intro.ipynb

@@ -88,41 +88,49 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Algorithm:\n",
-    "\n",
-    "Input: x: speech_sample, u:UBM, s:speaker_model\n",
+    "## Training UBM for VoxForge dataset\n",
     "\n",
-    "1. Define a threshold c2.\n",
-    "2. Compute F: array of MFCC features from x\n",
-    "2. Compute A = ln(p(F|speaker_model)): log(probability that speech-sample was produced by speaker-model of claimed-identity)\n",
-    "3. Compute B = ln(p(F|UBM)): log(probability that speech was produced by some other person in the world)\n",
-    "4. Return score = (A - B)\n"
+    "![](figures/asv_training.png)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Training UBM for VoxForge dataset\n",
+    "## Enrollment Procedure\n",
     "\n",
-    "![](figures/asv_training.png)"
+    "Input: x: speech_sample, u: UBM, i: identity\n",
+    "\n",
+    "1. Compute F: array of MFCC features from x\n",
+    "2. Apply MAP adaptation on UBM_GMM to generate Speaker_GMM\n",
+    "3. Store Speaker_GMM model for Speaker-i.\n",
+    "\n",
+    "\n",
+    "![](figures/asv_enrollment.png)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Given:\n",
+    "## Probe Algorithm:\n",
     "\n",
-    "1. Pre-computed features: one feature-set per presentation audio.\n",
-    "2. Pre-computed UBM: GMM with 64 components\n",
-    "3. Pre-trained threshold, determined using 'dev' set.\n",
-    "4. Pre-enrolled speakers in 'eval' set.\n",
+    "Input: x: speech_sample, u: UBM, s: Speaker_model of 'claimed identity'\n",
     "\n",
-    "## To do:\n",
-    "1. For each enrolled speaker:\n",
-    "    2. For feature-set of every probe-sample in 'eval' set:\n",
-    "        3. Determine if sample came from enrolled speaker"
+    "1. Define a threshold c2.\n",
+    "2. Compute F: array of MFCC features from x\n",
+    "2. Compute A = ln(p(F|speaker_model)): log(probability that speech-sample was produced by speaker-model of claimed-identity)\n",
+    "3. Compute B = ln(p(F|UBM)): log(probability that speech was produced by some other person in the world)\n",
+    "4. Return score = (A - B)\n",
+    "\n",
+    "![](figures/asv_probing.png)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Select Threshold Using 'Dev' Set"
    ]
   },
   {
@@ -131,6 +139,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "%matplotlib inline\n",
     "import os\n",
     "\n",
     "code_dir = os.getcwd()\n",
@@ -152,15 +161,18 @@
     "import re, sys\n",
     "import numpy\n",
     "from pathlib import Path\n",
+    "from speaker_lib import load_scores\n",
     "\n",
-    "my_file_name = \"data/voxforge_denoised_scores_dev.txt\"\n",
-    "%print(\"My file is %s\" %my_file_name)\n",
-    "my_file = Path(my_file_name)\n",
-    "assert my_file.is_file(), \"File %s does not exist. Quitting.\" %my_file_name\n",
+    "my_file_name = \"data/voxforge_denoised_16K_scores_dev.txt\"\n",
+    "#print(\"My file is %s\" %my_file_name)\n",
+    "#my_file = Path(my_file_name)\n",
+    "#assert my_file.is_file(), \"File %s does not exist. Quitting.\" %my_file_name\n",
+    "#\n",
+    "#dsf = open(\"data/voxforge_denoised_16K_scores_dev.txt\", \"r\")\n",
+    "#x = dsf.readlines()\n",
+    "#dsf.close()\n",
     "\n",
-    "dsf = open(\"data/voxforge_denoised_scores_dev.txt\", \"r\")\n",
-    "x = dsf.readlines()\n",
-    "dsf.close()\n",
+    "dev_zei_scores, dev_gen_scores = load_scores(my_file_name)\n",
     "\n",
     "dev_gen_scores = [float(line.split()[3]) for line in x if line.split()[0] == line.split()[1]]\n",
     "dev_zei_scores = [float(line.split()[3]) for line in x if line.split()[0] != line.split()[1]]\n",
@@ -173,7 +185,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Score Distribution for 'dev' set"
+    "## Plot Score Distribution for 'dev' set\n",
+    "\n",
+    "Green: histogram of scores of genuine presentations\n",
+    "Blue: histogram of scores of zero-effort-impostor presentations "
    ]
   },
   {
@@ -193,6 +208,19 @@
     "print(\"Threshold for FMR=0.01% on dev-set:\" '{:6.2f}'.format(dev_fmr_001))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Plot the Detection Error Tradeoff (DET) Curve\n",
+    "\n",
+    "There are two kinds of detection error:\n",
+    " - Genuine presentation classified as Impostor: False non-match rate (FNMR)\n",
+    " - Impostor presentation classified as Genuine: False Match Rate (FMR)\n",
+    "\n",
+    "DET curve shows how much FNMR you should expect as you adjust the Threshold (left to right) to achieve a desired FMR."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -202,6 +230,23 @@
     "plots.plot_det(dev_zero_effort_impostor_scores, dev_genuine_scores)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Given:\n",
+    "\n",
+    "1. Pre-computed features: one feature-set per presentation audio.\n",
+    "2. Pre-computed UBM: GMM with 64 components\n",
+    "3. Pre-trained threshold, determined using 'dev' set.\n",
+    "4. Pre-enrolled speakers in 'eval' set.\n",
+    "\n",
+    "## To do:\n",
+    "1. For each enrolled speaker:\n",
+    "    2. For feature-set of every probe-sample in 'eval' set:\n",
+    "        3. Determine if sample came from enrolled speaker"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

notebooks/model.08_asv_expts.ipynb

@@ -78,22 +78,31 @@
    "outputs": [],
    "source": [
     "from pathlib import Path\n",
+    "import numpy\n",
+    "from speaker_lib import load_scores\n",
     "\n",
-    "my_file_name = \"data/voxforge_denoised_scores_dev.txt\"\n",
-    "my_file = Path(my_file_name)\n",
-    "assert my_file.is_file(), \"File %s does not exist. Quitting.\" %my_file_name\n",
-    "\n",
-    "dsf = open(\"data/voxforge_denoised_scores_dev.txt\", \"r\")\n",
-    "x = dsf.readlines()\n",
-    "dsf.close()\n",
-    "\n",
-    "dev_gen_scores = [float(line.split()[3]) for line in x if line.split()[0] == line.split()[1]]\n",
-    "dev_zei_scores = [float(line.split()[3]) for line in x if line.split()[0] != line.split()[1]]\n",
+    "my_file_name = \"data/voxforge_denoised_16K_scores_dev.txt\"\n",
+    "# we assume that the file is in the 4-column format devised for bob-score-files.\n",
     "\n",
-    "dev_genuine_scores = numpy.array(dev_gen_scores)\n",
-    "dev_zero_effort_imposter_scores = numpy.array(dev_zei_scores)\n",
-    "dev_fmr_01 = bob.measure.far_threshold (dev_zero_effort_imposter_scores, dev_genuine_scores, 0.1)\n",
-    "dev_fmr_001 = bob.measure.far_threshold(dev_zero_effort_imposter_scores, dev_genuine_scores, 0.01)\n",
+    "#def load_scores(score_filename):\n",
+    "#    my_file = Path(score_filename)\n",
+    "#    assert my_file.is_file(), \"File %s does not exist. Quitting.\" %score_filename\n",
+    "#    \n",
+    "#    dsf = open(score_filename, \"r\")\n",
+    "#    x = dsf.readlines()\n",
+    "#    dsf.close()\n",
+    "#    \n",
+    "#    gen_scores = [float(line.split()[3]) for line in x if line.split()[0] == line.split()[1]]\n",
+    "#    zei_scores = [float(line.split()[3]) for line in x if line.split()[0] != line.split()[1]]\n",
+    "#    \n",
+    "#    return numpy.array(zei_scores), numpy.array(gen_scores)\n",
+    "    \n",
+    "dev16K_zei_scores, dev16K_gen_scores = load_scores(my_file_name)\n",
+    "#genuine_scores = numpy.array(dev16K_gen_scores)\n",
+    "#zero_effort_imposter_scores = numpy.array(dev16K_zei_scores)\n",
+    "dev_fmr_01 = bob.measure.far_threshold (dev16K_zei_scores, dev16K_gen_scores, 0.1)\n",
+    "dev_fmr_001 = bob.measure.far_threshold(dev16K_zei_scores, dev16K_gen_scores, 0.01)\n",
+    "    \n",
     "print(\"Thresholds from dev set:\")\n",
     "print(\"FMR @  0.1%:\" '{:6.2f}'.format(dev_fmr_01))\n",
     "print(\"FMR @ 0.01%:\" '{:6.2f}'.format(dev_fmr_001))"
@@ -164,7 +173,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Plotting score distr#ibution\n",
+    "# group the score-distributions, for ease of manipulation...\n",
     "zero_effort_scores = []\n",
     "genuine_scores = []\n",
     "for n in client_list:\n",
@@ -173,8 +182,26 @@
     "\n",
     "eval_genuine_scores = numpy.array(genuine_scores)\n",
     "eval_zero_effort_scores = numpy.array(zero_effort_scores)\n",
-    "# plot_hist(zero_effort_scores, genuine_scores, fmr_01=None, fmr_001=None):\n",
-    "plots.plot_hist(eval_zero_effort_scores, eval_genuine_scores, fmr_01=dev_fmr_01, fmr_001=dev_fmr_001)"
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Plot the score-distributions for genuine and ZEI presentations in 'eval' set\n",
+    "\n",
+    "The vertical lines represent the score-thresholds obtained from the 'dev' set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "#plots.plot_hist(eval_zero_effort_scores, eval_genuine_scores, fmr_01=dev_fmr_01, fmr_001=dev_fmr_001)"
    ]
   },
   {
@@ -183,6 +210,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "print(len(eval_zero_effort_scores))\n",
+    "print(len(eval_genuine_scores))\n",
     "plots.plot_roc(eval_zero_effort_scores, eval_genuine_scores)"
    ]
   },
@@ -203,6 +232,32 @@
    "source": [
     "add epc"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "score_file_list = ['scores_vf_denoised_8k_dev.txt', 'voxforge_denoised_16K_scores_dev.txt']\n",
+    "labels = ['8K', '16K']\n",
+    "plots.plot_multidet(score_file_list, labels, base_path=\"./data\")\n",
+    "plots.plot_multiroc(score_file_list, labels, base_path=\"./data\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {