Commit 5f5a74fa authored by Laurent EL SHAFEY's avatar Laurent EL SHAFEY

Initial port of the BANCA database as a satellite package for bob

parents
*~
*.swp
*.pyc
bin
eggs
parts
.installed.cfg
.mr.developer.cfg
*.egg-info
src
develop-eggs
built-docs
dist
sphinx
db.sql3
include README.rst
recursive-include docs *.py *.rst
recursive-include xbob *.sql3
================
BANCA Database
================
This package contains the access API and descriptions for the `BANCA
Database <http://www.ee.surrey.ac.uk/CVSSP/banca/>`_. The actual raw data for
the database should be downloaded from the original URL. This package only
contains the `Bob <http://www.idiap.ch/software/bob/>`_ accessor methods to use
the DB directly from python, with our certified protocols.
You would normally not install this package unless you are maintaining it. What
you would do instead is to tie it in at the package you need to **use** it.
There are a few ways to achieve this:
1. You can add this package as a requirement at the ``setup.py`` for your own
`satellite package
<https://github.com/idiap/bob/wiki/Virtual-Work-Environments-with-Buildout>`_
or to your Buildout ``.cfg`` file, if you prefer it that way. With this
method, this package gets automatically downloaded and installed on your
working environment, or
2. You can manually download and install this package using commands like
``easy_install`` or ``pip``.
The package is available in two different distribution formats:
1. You can download it from `PyPI <http://pypi.python.org/pypi>`_, or
2. You can download it in its source form from `its git repository
<https://github.com/bioidiap/xbob.db.banca>`_. When you download the
version at the git repository, you will need to run a command to recreate
the backend SQLite file required for its operation. This means that the
database raw files must be installed somewhere in this case. With option
``a`` you can run in `dummy` mode and only download the raw data files for
the database once you are happy with your setup.
You can mix and match points 1/2 and a/b above based on your requirements. Here
are some examples:
Modify your setup.py and download from PyPI
===========================================
That is the easiest. Edit your ``setup.py`` in your satellite package and add
the following entry in the ``install_requires`` section (note: ``...`` means
`whatever extra stuff you may have in-between`, don't put that on your
script)::
install_requires=[
...
"xbob.db.banca",
],
Proceed normally with your ``boostrap/buildout`` steps and you should be all
set. That means you can now import the ``xbob.db.banca`` namespace into your scripts.
Modify your buildout.cfg and download from git
==============================================
You will need to add a dependence to `mr.developer
<http://pypi.python.org/pypi/mr.developer/>`_ to be able to install from our
git repositories. Your ``buildout.cfg`` file should contain the following
lines::
[buildout]
...
extensions = mr.developer
auto-checkout = *
eggs = bob
...
xbob.db.banca
[sources]
xbob.db.banca = git https://github.com/bioidiap/xbob.db.banca.git
...
This diff is collapsed.
; vim: set fileencoding=utf-8 :
; Andre Anjos <andre.anjos@idiap.ch>
; Mon 16 Apr 08:29:18 2012 CEST
[buildout]
parts = external tests sphinx python
develop = .
versions = versions
eggs = bob
xbob.db.banca
[versions]
;If you would like to pin-down the recipes package version so you are not
;bothered with eventual updates, do it here. Note that, by pinning the version
;of the package, you will be also excluded from bug fixes.
;xbob.buildout = 0.1
[external]
recipe = xbob.buildout:external
egg-directories = /idiap/group/torch5spro/nightlies/last/install/linux-x86_64-release/lib
[tests]
recipe = xbob.buildout:nose
eggs = ${buildout:eggs}
script = tests.py
[sphinx]
recipe = xbob.buildout:sphinx
eggs = ${buildout:eggs}
source = ${buildout:directory}/docs
build = ${buildout:directory}/sphinx
[python]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
interpreter = python
dependent-scripts = true
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Andre Anjos <andre.anjos@idiap.ch>
# Mon 13 Aug 2012 12:38:15 CEST
#
# Copyright (C) 2011-2012 Idiap Research Institute, Martigny, Switzerland
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import sys, os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.pngmath',
'sphinx.ext.ifconfig',
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'bob.sphinxext.plot', # ours add source copying to install directory
]
# The viewcode extension appeared only on Sphinx >= 1.0.0
import sphinx
if sphinx.__version__ >= "1.0":
extensions.append('sphinx.ext.viewcode')
# Always includes todos
todo_include_todos = True
# If we are on OSX, the 'dvipng' path maybe different
dvipng_osx = '/opt/local/libexec/texlive/binaries/dvipng'
if os.path.exists(dvipng_osx): pngmath_dvipng = dvipng_osx
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'BANCA Database (Bob API)'
import time
copyright = u'%s, Idiap Research Institute' % time.strftime('%Y')
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
from xbob.db.banca.driver import Interface
version = Interface().version()
# The full version, including alpha/beta/rc tags.
release = version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['**/links.rst']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
if sphinx.__version__ >= "1.0":
html_theme = 'nature'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = 'bob'
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
html_logo = ''
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
html_favicon = ''
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'bobdbdoc'
# -- Options for LaTeX output --------------------------------------------------
# The paper size ('letter' or 'a4').
latex_paper_size = 'a4'
# The font size ('10pt', '11pt' or '12pt').
latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'bobdbman.tex', u'Bob',
u'Biometrics Group, Idiap Research Institute', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
latex_logo = ''
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# Included after all input documents
rst_epilog = ''
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'bob', u'BANCA (Bob API) Documentation', [u'Idiap Research Institute'], 1)
]
# We want to remove all private (i.e. _. or __.__) members
# that are not in the list of accepted functions
accepted_private_functions = ['__call__']
def member_function_test(app, what, name, obj, skip, options):
# test if we have a private function
if len(name) > 1 and name[0] == '_':
# test if this private function should be allowed
if name not in accepted_private_functions:
# omit privat functions that are not in the list of accepted private functions
return True
else:
# test if the method is documented
if not hasattr(obj, '__doc__') or not obj.__doc__:
return True
# Skips selected members in auto-generated documentation. Unfortunately, old
# versions of Boost.Python will not generate a __self__ member for static
# methods and that screws-up Sphinx processing.
if sphinx.__version__ < "1.0":
# We have to remove objects that do not have a __self__ attribute set
import types
if isinstance(obj, types.BuiltinFunctionType) and \
not hasattr(obj, '__self__') and what == 'class':
app.warn("Skipping %s %s (no __self__)" % (what, name))
return True
return False
# Default processing flags for sphinx
autoclass_content = 'both'
autodoc_member_order = 'bysource'
autodoc_default_flags = ['members', 'undoc-members', 'special-members', 'inherited-members', 'show-inheritance']
def setup(app):
app.connect('autodoc-skip-member', member_function_test)
.. vim: set fileencoding=utf-8 :
.. Laurent El Shafey <laurent.el-shafey@idiap.ch>
.. Tue 28 Aug 2012 18:09:40 CEST
================
BANCA Database
================
.. automodule:: xbob.db.banca
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Laurent El Shafey <laurent.el-shafey@idiap.ch>
from setuptools import setup, find_packages
# The only thing we do in this file is to call the setup() function with all
# parameters that define our package.
setup(
name='xbob.db.banca',
version='1.0.0a1',
description='BANCA Database Access API for Bob',
url='http://github.com/bioidiap/xbob.db.banca',
license='GPLv3',
author='Laurent El Shafey',
author_email='laurent.el-shafey@idiap.ch',
long_description=open('README.rst').read(),
# This line is required for any distutils based packaging.
packages=find_packages(),
include_package_data=True,
zip_safe=False,
install_requires=[
'setuptools',
'bob', # base signal proc./machine learning library
],
namespace_packages = [
'xbob',
'xbob.db',
],
entry_points = {
# bob database declaration
'bob.db': [
'banca = xbob.db.banca.driver:Interface',
],
# bob unittest declaration
'bob.test': [
'banca = xbob.db.banca.test:BancaDatabaseTest',
],
},
classifiers = [
'Development Status :: 4 - Beta',
'Intended Audience :: Developers',
'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
'Natural Language :: English',
'Programming Language :: Python',
'Topic :: Scientific/Engineering :: Artificial Intelligence',
'Topic :: Database :: Front-Ends',
],
)
#see http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
__import__('pkg_resources').declare_namespace(__name__)
#see http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
__import__('pkg_resources').declare_namespace(__name__)
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Laurent El Shafey <laurent.el-shafey@idiap.ch>
"""The BANCA Database for face verification
"""
from .query import Database
__all__ = ['Database']
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Laurent El Shafey <laurent.el-shafey@idiap.ch>
"""Checks for installed files.
"""
import os
import sys
# Driver API
# ==========
def checkfiles(args):
"""Checks existence of files based on your criteria"""
from .query import Database
db = Database()
r = db.files(
directory=args.directory,
extension=args.extension,
protocol=args.protocol,
purposes=args.purposes,
#model_ids=args.model_ids,
groups=args.groups,
languages=args.languages
)
# go through all files, check if they are available on the filesystem
good = {}
bad = {}
for id, f in r.items():
if os.path.exists(f): good[id] = f
else: bad[id] = f
# report
output = sys.stdout
if args.selftest:
from bob.db.utils import null
output = null()
if bad:
for id, f in bad.items():
output.write('Cannot find file "%s"\n' % (f,))
output.write('%d files (out of %d) were not found at "%s"\n' % \
(len(bad), len(r), args.directory))
return 0
def add_command(subparsers):
"""Add specific subcommands that the action "checkfiles" can use"""
from argparse import SUPPRESS
parser = subparsers.add_parser('checkfiles', help=checkfiles.__doc__)
parser.add_argument('-d', '--directory', dest="directory", default='', help="if given, this path will be prepended to every entry returned (defaults to '%(default)s')")
parser.add_argument('-e', '--extension', dest="extension", default='', help="if given, this extension will be appended to every entry returned (defaults to '%(default)s')")
parser.add_argument('-p', '--protocol', dest="protocol", default='', help="if given, limits the check to a particular subset of the data that corresponds to the given protocol (defaults to '%(default)s')", choices=('P', 'G', 'Mc', 'Md', 'Ma', 'Ud', 'Ua', ''))
parser.add_argument('-u', '--purposes', dest="purposes", default='', help="if given, this value will limit the output files to those designed for the given purposes. (defaults to '%(default)s')", choices=('enrol', 'probe', ''))
# TODO: model_ids
parser.add_argument('-g', '--groups', dest="groups", default='', help="if given, this value will limit the output files to those belonging to a particular protocolar group. (defaults to '%(default)s')", choices=('g1', 'g2', 'world', ''))
parser.add_argument('-l', '--languages', dest="languages", default='', help="if given, this value will limit the output files to those belonging to the given languages. (defaults to '%(default)s')", choices=('en', ''))
parser.add_argument('-c', '--classes', dest="classes", default='', help="if given, this value will limit the output files to those belonging to the given classes. (defaults to '%(default)s')", choices=('client', 'impostor', ''))
parser.add_argument('--self-test', dest="selftest", default=False,
action='store_true', help=SUPPRESS)
parser.set_defaults(func=checkfiles) #action
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# Laurent El Shafey <Laurent.El-Shafey@idiap.ch>
# Fri 20 May 17:00:50 2011
"""This script creates the BANCA database in a single pass.
"""
import os
from .models import *
from bob.db.utils import session
def add_files(session, imagedir):
"""Add files (and clients) to the BANCA database."""
def add_file(session, filename, client_dict):
"""Parse a single filename and add it to the list.
Also add a client entry if not already in the database."""
v = os.path.splitext(os.path.basename(filename))[0].split('_')
if not (v[0] in client_dict):
if (v[2] == 'wm'):
v[2] = 'world'
session.add(Client(int(v[0]), v[1], v[2], v[5]))
client_dict[v[0]] = True
session_id = int(v[3].split('s')[1])
session.add(File(int(v[0]), os.path.basename(filename).split('.')[0], v[4], v[6], session_id))
file_list = os.listdir(imagedir)
client_dict = {}
for filename in file_list:
add_file(session, os.path.join(imagedir, filename), client_dict)
def add_subworlds(session):
"""Adds splits in the world set, based on the client ids"""
# one third
session.add(Subworld("onethird", 9003))
session.add(Subworld("onethird", 9005))
session.add(Subworld("onethird", 9027))
session.add(Subworld("onethird", 9033))
session.add(Subworld("onethird", 9035))
session.add(Subworld("onethird", 9043))
session.add(Subworld("onethird", 9049))
session.add(Subworld("onethird", 9053))
session.add(Subworld("onethird", 9055))
session.add(Subworld("onethird", 9057))
# two thirds
session.add(Subworld("twothirds", 9001))
session.add(Subworld("twothirds", 9007))
session.add(Subworld("twothirds", 9009))
session.add(Subworld("twothirds", 9011))
session.add(Subworld("twothirds", 9013))
session.add(Subworld("twothirds", 9015))
session.add(Subworld("twothirds", 9017))
session.add(Subworld("twothirds", 9019))
session.add(Subworld("twothirds", 9021))
session.add(Subworld("twothirds", 9023))
session.add(Subworld("twothirds", 9025))
session.add(Subworld("twothirds", 9029))
session.add(Subworld("twothirds", 9031))
session.add(Subworld("twothirds", 9037))
session.add(Subworld("twothirds", 9039))
session.add(Subworld("twothirds", 9041))
session.add(Subworld("twothirds", 9045))
session.add(Subworld("twothirds", 9047))
session.add(Subworld("twothirds", 9051))
session.add(Subworld("twothirds", 9059))
def add_sessions(session):
"""Adds relations between sessions and scenarios"""
for i in range(1,5):
session.add(Session(i,'controlled'))
for i in range(5,9):
session.add(Session(i,'degraded'))
for i in range(9,13):
session.add(Session(i,'adverse'))
def add_protocols(session):
"""Adds protocols"""
# Protocol Mc
session.add(Protocol(1, 'Mc', 'enrol'))
session.add(Protocol(1, 'Mc', 'probeImpostor'))
session.add(Protocol(2, 'Mc', 'probe'))
session.add(Protocol(3, 'Mc', 'probe'))
session.add(Protocol(4, 'Mc', 'probe'))
# Protocol Md
session.add(Protocol(5, 'Md', 'enrol'))
session.add(Protocol(5, 'Md', 'probeImpostor'))
session.add(Protocol(6, 'Md', 'probe'))
session.add(Protocol(7, 'Md', 'probe'))
session.add(Protocol(8, 'Md', 'probe'))
# Protocol Ma
session.add(Protocol(9, 'Ma', 'enrol'))
session.add(Protocol(9, 'Ma', 'probeImpostor'))
session.add(Protocol(10, 'Ma', 'probe'))
session.add(Protocol(11, 'Ma', 'probe'))
session.add(Protocol(12, 'Ma', 'probe'))
# Protocol Ud
session.add(Protocol(1, 'Ud', 'enrol'))
session.add(Protocol(5, 'Ud', 'probeImpostor'))
session.add(Protocol(6, 'Ud', 'probe'))
session.add(Protocol(7, 'Ud', 'probe'))
session.add(Protocol(8, 'Ud', 'probe'))
# Protocol Ma
session.add(Protocol(1, 'Ua', 'enrol'))
session.add(Protocol(9, 'Ua', 'probeImpostor'))
session.add(Protocol(10, 'Ua', 'probe'))
session.add(Protocol(11, 'Ua', 'probe'))
session.add(Protocol(12, 'Ua', 'probe'))
# Protocol P
session.add(Protocol(1, 'P', 'enrol'))
session.add(Protocol(1, 'P', 'probeImpostor'))
session.add(Protocol(2, 'P', 'probe'))
session.add(Protocol(3, 'P', 'probe'))
session.add(Protocol(4, 'P', 'probe'))
session.add(Protocol(5, 'P', 'probeImpostor'))
session.add(Protocol(6, 'P', 'probe'))
session.add(Protocol(7, 'P', 'probe'))
session.add(Protocol(8, 'P', 'probe'))
session.add(Protocol(9, 'P', 'probeImpostor'))
session.add(Protocol(10, 'P', 'probe'))
session.add(Protocol(11, 'P', 'probe'))
session.add(Protocol(12, 'P', 'probe'))
# Protocol G
session.add(Protocol(1, 'G', 'enrol'))
session.add(Protocol(5, 'G', 'enrol'))
session.add(Protocol(9, 'G', 'enrol'))
session.add(Protocol(1, 'G', 'probeImpostor'))
session.add(Protocol(2, 'G', 'probe'))
session.add(Protocol(3, 'G', 'probe'))
session.add(Protocol(4, 'G', 'probe'))
session.add(Protocol(5, 'G', 'probeImpostor'))
session.add(Protocol(6, 'G', 'probe'))
session.add(Protocol(7, 'G', 'probe'))
session.add(Protocol(8, 'G', 'probe'))
session.add(Protocol(9, 'G', 'probeImpostor'))
session.add(Protocol(10, 'G', 'probe'))
session.add(Protocol(11, 'G', 'probe'))
session.add(Protocol(12, 'G', 'probe'))
def create_tables(args):
"""Creates all necessary tables (only to be used at the first time)"""
from bob.db.utils import connection_string