Skip to content
Snippets Groups Projects
Commit f4c7d6ca authored by Amir MOHAMMADI's avatar Amir MOHAMMADI
Browse files

Merge branch 'pre-release' into 'master' 

Pre release

Documenting and fixing warnings
and closes #41 

See merge request !42
parents 303ed8f2 104407b3
No related branches found
No related tags found
1 merge request!42Pre release
Pipeline #
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 547 additions and 507 deletions
bob/bio/base/algorithm/Algorithm.py
+ 1
− 1
View file @ f4c7d6ca
......@@ -47,7 +47,7 @@ class Algorithm:
See :py:func:`bob.bio.base.score_fusion_strategy` for possible values.
kwargs : ``key=value`` pairs
A list of keyword arguments to be written in the :py:meth:`__str__` function.
A list of keyword arguments to be written in the `__str__` function.
"""
......
bob/bio/base/algorithm/BIC.py
+ 234
− 225
View file @ f4c7d6ca
This diff is collapsed.
bob/bio/base/database/database.py
+ 83
− 78
View file @ f4c7d6ca
......@@ -10,7 +10,64 @@ import bob.db.base
import bob.bio.base.database
class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
"""This class represents the basic API for database access.
Please use this class as a base class for your database access classes.
Do not forget to call the constructor of this base class in your derived class.
**Parameters:**
name : str
A unique name for the database.
all_files_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query when retrieving all data.
extractor_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the extractor training.
projector_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the projector training.
enroller_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the enroller training.
check_original_files_for_existence : bool
Enables to test for the original data files when querying the database.
original_directory : str
The directory where the original data of the database are stored.
original_extension : str
The file name extension of the original data.
annotation_directory : str
The directory where the image annotations of the database are stored, if any.
annotation_extension : str
The file name extension of the annotation files.
annotation_type : str
The type of the annotation file to read, see `bob.db.base.annotations.read_annotation_file` for accepted formats.
protocol : str or ``None``
The name of the protocol that defines the default experimental setup for this database.
.. todo:: Check if the ``None`` protocol is supported.
training_depends_on_protocol : bool
Specifies, if the training set used for training the extractor and the projector depend on the protocol.
This flag is used to avoid re-computation of data when running on the different protocols of the same database.
models_depend_on_protocol : bool
Specifies, if the models depend on the protocol.
This flag is used to avoid re-computation of models when running on the different protocols of the same database.
kwargs : ``key=value`` pairs
The arguments of the :py:class:`Database` base class constructor.
"""
def __init__(
self,
name,
......@@ -32,62 +89,6 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
models_depend_on_protocol=False,
**kwargs
):
"""This class represents the basic API for database access.
Please use this class as a base class for your database access classes.
Do not forget to call the constructor of this base class in your derived class.
**Parameters:**
name : str
A unique name for the database.
all_files_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query when retrieving all data.
extractor_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the extractor training.
projector_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the projector training.
enroller_training_options : dict
Dictionary of options passed to the :py:meth:`bob.bio.base.database.BioDatabase.objects` database query used to retrieve the files for the enroller training.
check_original_files_for_existence : bool
Enables to test for the original data files when querying the database.
original_directory : str
The directory where the original data of the database are stored.
original_extension : str
The file name extension of the original data.
annotation_directory : str
The directory where the image annotations of the database are stored, if any.
annotation_extension : str
The file name extension of the annotation files.
annotation_type : str
The type of the annotation file to read, see :py:func:`bob.bio.base.database.read_annotation_file` for accepted formats.
protocol : str or ``None``
The name of the protocol that defines the default experimental setup for this database.
.. todo:: Check if the ``None`` protocol is supported.
training_depends_on_protocol : bool
Specifies, if the training set used for training the extractor and the projector depend on the protocol.
This flag is used to avoid re-computation of data when running on the different protocols of the same database.
models_depend_on_protocol : bool
Specifies, if the models depend on the protocol.
This flag is used to avoid re-computation of models when running on the different protocols of the same database.
kwargs : ``key=value`` pairs
The arguments of the :py:class:`Database` base class constructor.
"""
assert isinstance(name, str)
......@@ -214,13 +215,13 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Parameters:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The list of files to be uniquified and sorted.
**Returns:**
sorted : [:py:class:`File`]
The sorted list of files, with duplicate :py:attr:`File.id`\s being removed.
sorted : [:py:class:`BioFile`]
The sorted list of files, with duplicate `BioFile.id`\s being removed.
"""
# sort files using their sort function
sorted_files = sorted(files)
......@@ -242,13 +243,13 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Parameters:**
files : :py:class:`File`
A list of files that should be split up by :py:attr:`File.client_id`.
files : :py:class:`BioFile`
A list of files that should be split up by `BioFile.client_id`.
**Returns:**
files_by_client : [[:py:class:`File`]]
The list of lists of files, where each sub-list groups the files with the same :py:attr:`File.client_id`
files_by_client : [[:py:class:`BioFile`]]
The list of lists of files, where each sub-list groups the files with the same `BioFile.client_id`
"""
client_files = {}
for file in files:
......@@ -264,11 +265,11 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
def annotations(self, file):
"""
Returns the annotations for the given File object, if available.
It uses :py:func:`bob.bio.base.database.read_annotation_file` to load the annotations.
It uses `bob.db.base.annotations.read_annotation_file` to load the annotations.
**Parameters:**
file : :py:class:`File`
file : :py:class:`BioFile`
The file for which annotations should be returned.
**Returns:**
......@@ -293,7 +294,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Parameters:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The list of file object to retrieve the file names for.
directory : str
......@@ -323,7 +324,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Parameters:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The list of file object to retrieve the original data file names for.
**Returns:**
......@@ -413,7 +414,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
Keyword parameters:
file : :py:class:`File` or a derivative
file : :py:class:`BioFile` or a derivative
The File objects for which the file name should be retrieved
Return value : str
......@@ -444,7 +445,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The sorted and unique list of all files of the database.
"""
return self.sort(self.objects(protocol=self.protocol, groups=groups, **self.all_files_options))
......@@ -467,7 +468,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Returns:**
files : [:py:class:`File`] or [[:py:class:`File`]]
files : [:py:class:`BioFile`] or [[:py:class:`BioFile`]]
The (arranged) list of files used for the training of the given step.
"""
if step is None:
......@@ -502,7 +503,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The sorted and unique list of test files of the database.
"""
return self.sort(self.objects(protocol=self.protocol, groups=groups, **self.all_files_options))
......@@ -551,7 +552,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The list of files used for to probe the model with the given model id.
"""
if model_id is not None:
......@@ -605,7 +606,7 @@ class BioDatabase(six.with_metaclass(abc.ABCMeta, bob.db.base.Database)):
**Returns:**
files : [:py:class:`FileSet`] or something similar
files : [:py:class:`BioFileSet`] or something similar
The list of file sets used to probe the model with the given model id."""
if model_id is not None:
file_sets = self.object_sets(protocol=self.protocol, groups=group, model_ids=(model_id,), purposes='probe',
......@@ -714,7 +715,7 @@ class ZTBioDatabase(BioDatabase):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The sorted and unique list of all files of the database.
"""
files = self.objects(protocol=self.protocol, groups=groups, **self.all_files_options)
......@@ -776,7 +777,7 @@ class ZTBioDatabase(BioDatabase):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The sorted list of files used for to enroll the model with the given model id.
"""
return self.sort(self.tobjects(protocol=self.protocol, groups=group, model_ids=(t_model_id,)))
......@@ -785,7 +786,7 @@ class ZTBioDatabase(BioDatabase):
"""z_probe_files(group = 'dev') -> files
Returns a list of probe files used to compute the Z-Norm, respecting the current protocol.
The Z-probe files can be limited using the ``z_probe_options`` in the query to :py:meth:`ZTBioDatabase.z_probe_files`
The Z-probe files can be limited using the ``z_probe_options`` in the query to :py:meth:`bob.bio.base.database.ZTBioDatabase.z_probe_files`
**Parameters:**
......@@ -794,7 +795,7 @@ class ZTBioDatabase(BioDatabase):
**Returns:**
files : [:py:class:`File`]
files : [:py:class:`BioFile`]
The unique list of files used to compute the Z-norm.
"""
return self.sort(self.zobjects(protocol=self.protocol, groups=group, **self.z_probe_options))
......@@ -812,7 +813,7 @@ class ZTBioDatabase(BioDatabase):
**Returns:**
files : [:py:class:`FileSet`]
files : [:py:class:`BioFileSet`]
The unique list of file sets used to compute the Z-norm.
"""
raise NotImplementedError("Please implement this function in derived classes")
......@@ -822,12 +823,16 @@ class ZTBioDatabase(BioDatabase):
Returns the client id for the given T-Norm model id.
In this base class implementation, we just use the :py:meth:`client_id_from_model_id` function.
Overload this function if you need another behavior.
**Parameters:**
t_model_id : int or str
A unique ID that identifies the T-Norm model.
group : one of ``('dev', 'eval')``
The group to get the client ids for.
**Returns:**
client_id : [int] or [str]
A unique ID that identifies the client, to which the T-Norm model belongs.
"""
......
bob/bio/base/database/file.py
+ 34
− 32
View file @ f4c7d6ca
......@@ -18,7 +18,7 @@ class BioFile(bob.db.base.File):
The id of the client this file belongs to.
Its type depends on your implementation.
If you use an SQL database, this should be an SQL type like Integer or String.
For path and file_id, please refer to :py:class:`bob.db.base.File` constructor
For path and file_id, please refer to :py:class:`bob.db.base.file.File` constructor
"""
bob.db.base.File.__init__(self, path, file_id)
......@@ -28,34 +28,36 @@ class BioFile(bob.db.base.File):
class BioFileSet(BioFile):
"""This class defines the minimum interface of a set of database files that needs to be exported.
Use this class, whenever the database provides several files that belong to the same probe.
Each file set has an id, and a list of associated files, which are of type :py:class:`BioFile` of the same client.
The file set id can be anything hashable, but needs to be unique all over the database.
**Parameters:**
file_set_id : str or int
A unique ID that identifies the file set.
files : [:py:class:`BioFile`]
A non-empty list of BioFile objects that should be stored inside this file.
All files of that list need to have the same client ID.
"""
def __init__(self, file_set_id, files, path=None):
# don't accept empty file lists
assert len(files), "Cannot create an empty BioFileSet"
# call base class constructor
BioFile.__init__(self, files[0].client_id, "+".join(f.path for f in files) if path is None else path, file_set_id)
# check that all files come from the same client
assert all(f.client_id == self.client_id for f in files)
# The list of files contained in this set
self.files = files
"""The list of :py:class:`BioFile` objects stored in this file set"""
def __lt__(self, other):
"""Defines an order between file sets by using the order of the file set ids."""
# compare two BioFile set objects by comparing their IDs
return self.id < other.id
"""This class defines the minimum interface of a set of database files that needs to be exported.
Use this class, whenever the database provides several files that belong to the same probe.
Each file set has an id, and a list of associated files, which are of type :py:class:`BioFile` of the same client.
The file set id can be anything hashable, but needs to be unique all over the database.
**Parameters:**
file_set_id : str or int
A unique ID that identifies the file set.
files : [:py:class:`BioFile`]
A non-empty list of BioFile objects that should be stored inside this file.
All files of that list need to have the same client ID.
"""
def __init__(self, file_set_id, files, path=None):
# don't accept empty file lists
assert len(files), "Cannot create an empty BioFileSet"
# call base class constructor
BioFile.__init__(self, files[0].client_id, "+".join(f.path for f in files) if path is None else path,
file_set_id)
# check that all files come from the same client
assert all(f.client_id == self.client_id for f in files)
# The list of files contained in this set
self.files = files
"""The list of :py:class:`BioFile` objects stored in this file set"""
def __lt__(self, other):
"""Defines an order between file sets by using the order of the file set ids."""
# compare two BioFile set objects by comparing their IDs
return self.id < other.id
bob/bio/base/extractor/Extractor.py
+ 2
− 2
View file @ f4c7d6ca
......@@ -24,7 +24,7 @@ class Extractor:
Ignored, if ``requires_training`` is ``False``
kwargs : ``key=value`` pairs
A list of keyword arguments to be written in the :py:meth:`__str__` function.
A list of keyword arguments to be written in the `__str__` function.
"""
def __init__(
......@@ -89,7 +89,7 @@ class Extractor:
**Parameters:**
feature : object
The extracted feature, i.e., what is returned from :py:meth:`__call__`.
The extracted feature, i.e., what is returned from `__call__`.
feature_file : str or :py:class:`bob.io.base.HDF5File`
The file open for writing, or the name of the file to write.
......
bob/bio/base/extractor/Linearize.py
+ 27
− 26
View file @ f4c7d6ca
......@@ -6,40 +6,41 @@
from .Extractor import Extractor
import numpy
class Linearize (Extractor):
"""Extracts features by simply concatenating all elements of the data into one long vector.
If a ``dtype`` is specified in the contructor, it is assured that the resulting
"""
class Linearize(Extractor):
"""Extracts features by simply concatenating all elements of the data into one long vector.
def __init__(self, dtype=None):
"""If the ``dtype`` parameter is given, it specifies the data type that is enforced for the features."""
Extractor.__init__(self, dtype = dtype)
self.dtype = dtype
If a ``dtype`` is specified in the contructor, it is assured that the resulting
"""
def __call__(self, data):
"""__call__(data) -> data
def __init__(self, dtype=None):
"""If the ``dtype`` parameter is given, it specifies the data type that is enforced for the features."""
Extractor.__init__(self, dtype=dtype)
self.dtype = dtype
Takes data of arbitrary dimensions and linearizes it into a 1D vector; enforcing the data type, if desired.
def __call__(self, data):
"""__call__(data) -> data
**Parameters:**
Takes data of arbitrary dimensions and linearizes it into a 1D vector; enforcing the data type, if desired.
data : :py:class:`numpy.ndarray`
The preprocessed data to be transformed into one vector.
**Parameters:**
**Returns:**
data : :py:class:`numpy.ndarray`
The preprocessed data to be transformed into one vector.
data : 1D :py:class:`numpy.ndarray`
The extracted feature vector, of the desired ``dtype`` (if specified).
"""
assert isinstance(data, numpy.ndarray)
**Returns:**
data : 1D :py:class:`numpy.ndarray`
The extracted feature vector, of the desired ``dtype`` (if specified).
"""
assert isinstance(data, numpy.ndarray)
linear = numpy.reshape(data, data.size)
if self.dtype is not None:
linear = linear.astype(self.dtype)
return linear
linear = numpy.reshape(data, data.size)
if self.dtype is not None:
linear = linear.astype(self.dtype)
return linear
# re-define unused functions, just so that they do not get documented
def train(*args, **kwargs): raise NotImplementedError()
# re-define unused functions, just so that they do not get documented
def train(*args,**kwargs): raise NotImplementedError()
def load(*args,**kwargs): pass
def load(*args, **kwargs): pass
bob/bio/base/preprocessor/Preprocessor.py
+ 70
− 73
View file @ f4c7d6ca
......@@ -5,104 +5,101 @@
from .. import utils
class Preprocessor:
"""This is the base class for all preprocessors.
It defines the minimum requirements for all derived proprocessor classes.
**Parameters:**
writes_data : bool
Select, if the preprocessor actually writes preprocessed images, or if it is simply returning values.
read_original_data: callable
This function is used to read the original data from file.
It takes three inputs: A :py:class:`bob.bio.base.database.BioFile` (or one of its derivatives), the original directory (as ``str``) and the original extension (as ``str``).
kwargs : ``key=value`` pairs
A list of keyword arguments to be written in the :py:meth:`__str__` function.
"""
class Preprocessor:
"""This is the base class for all preprocessors.
It defines the minimum requirements for all derived proprocessor classes.
def __init__(self, writes_data = True, read_original_data = utils.read_original_data, **kwargs):
# Each class needs to have a constructor taking
# all the parameters that are required for the preprocessing as arguments
self.writes_data = writes_data
self.read_original_data = read_original_data
self._kwargs = kwargs
pass
**Parameters:**
writes_data : bool
Select, if the preprocessor actually writes preprocessed images, or if it is simply returning values.
# The call function (i.e. the operator() in C++ terms)
def __call__(self, data, annotations):
"""__call__(data, annotations) -> dara
read_original_data: callable
This function is used to read the original data from file.
It takes three inputs: A :py:class:`bob.bio.base.database.BioFile` (or one of its derivatives), the original directory (as ``str``) and the original extension (as ``str``).
This is the call function that you have to overwrite in the derived class.
The parameters that this function will receive are:
kwargs : ``key=value`` pairs
A list of keyword arguments to be written in the `__str__` function.
"""
**Parameters:**
def __init__(self, writes_data=True, read_original_data=utils.read_original_data, **kwargs):
# Each class needs to have a constructor taking
# all the parameters that are required for the preprocessing as arguments
self.writes_data = writes_data
self.read_original_data = read_original_data
self._kwargs = kwargs
pass
data : object
The original data that needs preprocessing, usually a :py:class:`numpy.ndarray`, but might be different.
# The call function (i.e. the operator() in C++ terms)
def __call__(self, data, annotations):
"""__call__(data, annotations) -> dara
annotations : {} or None
The annotations (if any) that belongs to the given ``data``; as a dictionary.
The type of the annotation depends on your kind of problem.
This is the call function that you have to overwrite in the derived class.
The parameters that this function will receive are:
**Returns:**
**Parameters:**
data : object
The *preprocessed* data, usually a :py:class:`numpy.ndarray`, but might be different.
"""
raise NotImplementedError("Please overwrite this function in your derived class")
data : object
The original data that needs preprocessing, usually a :py:class:`numpy.ndarray`, but might be different.
annotations : {} or None
The annotations (if any) that belongs to the given ``data``; as a dictionary.
The type of the annotation depends on your kind of problem.
def __str__(self):
"""__str__() -> info
**Returns:**
This function returns all parameters of this class (and its derived class).
data : object
The *preprocessed* data, usually a :py:class:`numpy.ndarray`, but might be different.
"""
raise NotImplementedError("Please overwrite this function in your derived class")
**Returns:**
def __str__(self):
"""__str__() -> info
info : str
A string containing the full information of all parameters of this (and the derived) class.
"""
return utils.pretty_print(self, self._kwargs)
This function returns all parameters of this class (and its derived class).
############################################################
### Special functions that might be overwritten on need
############################################################
**Returns:**
info : str
A string containing the full information of all parameters of this (and the derived) class.
"""
return utils.pretty_print(self, self._kwargs)
def write_data(self, data, data_file):
"""Writes the given *preprocessed* data to a file with the given name.
In this base class implementation, we simply use :py:func:`bob.bio.base.save` for that.
If you have a different format (e.g. not images), please overwrite this function.
############################################################
### Special functions that might be overwritten on need
############################################################
**Parameters:**
def write_data(self, data, data_file):
"""Writes the given *preprocessed* data to a file with the given name.
In this base class implementation, we simply use :py:func:`bob.bio.base.save` for that.
If you have a different format (e.g. not images), please overwrite this function.
data : object
The preprocessed data, i.e., what is returned from :py:meth:`__call__`.
**Parameters:**
data_file : str or :py:class:`bob.io.base.HDF5File`
The file open for writing, or the name of the file to write.
"""
utils.save(data, data_file)
data : object
The preprocessed data, i.e., what is returned from `__call__`.
data_file : str or :py:class:`bob.io.base.HDF5File`
The file open for writing, or the name of the file to write.
"""
utils.save(data, data_file)
def read_data(self, data_file):
"""read_data(data_file) -> data
def read_data(self, data_file):
"""read_data(data_file) -> data
Reads the *preprocessed* data from file.
In this base class implementation, it uses :py:func:`bob.bio.base.load` to do that.
If you have different format, please overwrite this function.
Reads the *preprocessed* data from file.
In this base class implementation, it uses :py:func:`bob.bio.base.load` to do that.
If you have different format, please overwrite this function.
**Parameters:**
**Parameters:**
data_file : str or :py:class:`bob.io.base.HDF5File`
The file open for reading or the name of the file to read from.
data_file : str or :py:class:`bob.io.base.HDF5File`
The file open for reading or the name of the file to read from.
**Returns:**
**Returns:**
data : object (usually :py:class:`numpy.ndarray`)
The preprocessed data read from file.
"""
return utils.load(data_file)
data : object (usually :py:class:`numpy.ndarray`)
The preprocessed data read from file.
"""
return utils.load(data_file)
bob/bio/base/test/test_scripts.py
+ 0
− 1
View file @ f4c7d6ca
......@@ -144,7 +144,6 @@ def test_verify_parallel():
'-e', 'bob.bio.base.test.dummy.extractor.DummyExtractor()',
'-a', 'dummy',
'--zt-norm',
'--allow-missing-files',
'-vs', 'test_parallel',
'--temp-directory', test_dir,
'--result-directory', test_dir,
......
bob/bio/base/tools/FileSelector.py
+ 7
− 7
View file @ f4c7d6ca
......@@ -6,7 +6,7 @@ import os
from .. import utils
@utils.Singleton
class FileSelector:
class FileSelector(object):
"""This class provides shortcuts for selecting different files for different stages of the verification process.
It communicates with the database and provides lists of file names for all steps of the tool chain.
......@@ -15,7 +15,7 @@ class FileSelector:
**Parameters:**
database : :py:class:`bob.bio.base.database.Database` or derived
database : :py:class:`bob.bio.base.database.BioDatabase` or derived
The database object that provides the list of files.
preprocessed_directory : str
......@@ -107,12 +107,12 @@ class FileSelector:
### List of files that will be used for all files
def original_data_list(self, groups = None):
"""Returns the list of original data that can be used for preprocessing."""
return self.database.original_file_names(self.database.all_files(groups=groups))
"""Returns the list of original ``BioFile`` objects that can be used for preprocessing."""
return self.database.all_files(groups=groups)
def original_data_list_files(self, groups = None):
"""Returns the list of original data that can be used for preprocessing."""
return (self.database.all_files(groups=groups), self.database.original_directory, self.database.original_extension)
def original_directory_and_extension(self):
"""Returns the directory and extension of the original files."""
return self.database.original_directory, self.database.original_extension
def annotation_list(self, groups = None):
"""Returns the list of annotations objects."""
......
bob/bio/base/tools/command_line.py
+ 1
− 1
View file @ f4c7d6ca
......@@ -8,7 +8,6 @@ logger = bob.core.log.setup("bob.bio.base")
from .. import utils
from . import FileSelector
from bob.bio.base.database import BioDatabase
"""Execute biometric recognition algorithms on a certain biometric database.
"""
......@@ -201,6 +200,7 @@ def initialize(parsers, command_line_parameters = None, skips = []):
.. note:: The database, preprocessor, extractor, algorithm and grid (if specified) are actual instances of the according classes.
"""
from bob.bio.base.database import BioDatabase
# add execute-only flags to command line options
if skips is not None:
......
bob/bio/base/tools/preprocessor.py
+ 2
− 1
View file @ f4c7d6ca
......@@ -42,7 +42,8 @@ def preprocess(preprocessor, groups = None, indices = None, allow_missing_files
fs = FileSelector.instance()
# get the file lists
data_files, original_directory, original_extension = fs.original_data_list_files(groups=groups)
data_files = fs.original_data_list(groups=groups)
original_directory, original_extension = fs.original_directory_and_extension()
preprocessed_data_files = fs.preprocessed_data_list(groups=groups)
# select a subset of keys to iterate
......
bob/bio/base/utils/singleton.py
+ 3
− 2
View file @ f4c7d6ca
# A singleton class decorator, based on http://stackoverflow.com/a/7346105/3301902
class Singleton:
class Singleton(object):
"""
A non-thread-safe helper class to ease implementing singletons.
This should be used as a **decorator** -- not a metaclass -- to the class that should be a singleton.
The decorated class can define one `__init__` function that takes an arbitrary list of parameters.
To get the singleton instance, use the :py:meth:`instance` method. Trying to use :py:meth:`__call__` will result in a :py:class:`TypeError` being raised.
To get the singleton instance, use the :py:meth:`instance` method. Trying to use `__call__` will result in a `TypeError` being raised.
Limitations:
......@@ -21,6 +21,7 @@ class Singleton:
self.__doc__ = decorated.__doc__
self.__name__ = decorated.__name__
self.__module__ = decorated.__module__
self.__mro__ = decorated.__mro__
self.__bases__ = []
self._instance = None
......
doc/conf.py
+ 7
− 2
View file @ f4c7d6ca
......@@ -216,8 +216,13 @@ autodoc_default_flags = [
]
# For inter-documentation mapping:
from bob.extension.utils import link_documentation
intersphinx_mapping = link_documentation()
from bob.extension.utils import link_documentation, load_requirements
sphinx_requirements = "./extra-intersphinx.txt"
if os.path.exists(sphinx_requirements):
intersphinx_mapping = link_documentation(additional_packages=load_requirements(sphinx_requirements))
else:
intersphinx_mapping = link_documentation()
# We want to remove all private (i.e. _. or __.__) members
# that are not in the list of accepted functions
......
doc/extra-intersphinx.txt 0 → 100644
+ 11
− 0
View file @ f4c7d6ca
python
numpy
bob.bio.face
bob.bio.speaker
bob.bio.gmm
bob.bio.video
bob.bio.csu
bob.bio.spear
bob.learn.linear
gridtk
bob.db.youtube
\ No newline at end of file
doc/implementation.rst
+ 38
− 37
View file @ f4c7d6ca
......@@ -196,7 +196,6 @@ Here is a list of implementations:
* :ref:`bob.bio.face <bob.bio.face>` : :ref:`bob.bio.face.implemented`
* :ref:`bob.bio.video <bob.bio.video>` : :ref:`bob.bio.video.implemented`
* :ref:`bob.bio.gmm <bob.bio.gmm>` : :ref:`bob.bio.gmm.implemented`
* :ref:`bob.bio.csu <bob.bio.csu>` : :ref:`bob.bio.csu.implemented`
.. * :ref:`bob.bio.spear <bob.bio.spear>` : :ref:`bob.bio.spear.implemented`
......@@ -218,56 +217,58 @@ Verification Database Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For most of the data sets, we rely on the database interfaces from Bob_.
Particularly, all databases that are derived from the :py:class:`bob.bio.base.database.BioDatabase` (click :ref:`here <verification_databases>` for a list of implemented databases) are supported by a special derivation of the databases from above.
For these databases, the special :py:class:`bob.bio.base.database.DatabaseBob` interface is provided, which takes the Bob_ database as parameter.
Several such databases are defined in the according packages, i.e., :ref:`bob.bio.spear <bob.bio.spear>`, :ref:`bob.bio.face <bob.bio.face>` and :ref:`bob.bio.video <bob.bio.video>`.
For Bob_'s ZT-norm databases, we provide the :py:class:`bob.bio.base.database.DatabaseBobZT` interface.
Additionally, a generic database interface, which is derived from :py:class:`bob.bio.base.database.DatabaseBobZT`, is the :py:class:`bob.bio.base.database.DatabaseFileList`.
This database interfaces with the :py:class:`bob.db.verification.filelist.Database`, which is a generic database based on file lists, implementing the :py:class:`bob.bio.base.database.BioDatabase` interface.
Particularly, all databases that are derived from the :py:class:`bob.bio.base.database.BioDatabase` (click `here <https://github.com/idiap/bob/wiki/Packages>`_ for a list of implemented databases) are supported by a special derivation of the databases from above.
For these databases, the special :py:class:`bob.bio.base.database.BioDatabase` interface is provided, which takes the Bob_ database as parameter.
Several such databases are defined in the according packages, i.e., :ref:`bob.bio.spear <bob.bio.spear>`, :ref:`bob.bio.face <bob.bio.face>` and :ref:`bob.bio.video <bob.bio.video>`.
For Bob_'s ZT-norm databases, we provide the :py:class:`bob.bio.base.database.ZTBioDatabase` interface.
Defining your own Database
~~~~~~~~~~~~~~~~~~~~~~~~~~
..
If you have your own database that you want to execute the recognition experiments on, you should first check if you could use the :ref:`Verifcation File List Database <bob.db.bio_filelist>` interface by defining appropriate file lists for the training set, the model set, and the probes.
In most of the cases, the :py:class:`bob.db.bio_filelist.Database` should be sufficient to run experiments.
Please refer to the documentation :ref:`Documentation <bob.db.bio_filelist>` of this database for more instructions on how to configure this database.
If you have your own database that you want to execute the recognition experiments on, you should first check if you could use the :ref:`Verifcation FileList Database <bob.db.verification.filelist>` interface by defining appropriate file lists for the training set, the model set, and the probes.
In most of the cases, the :py:class:`bob.db.verification.filelist.Database` should be sufficient to run experiments.
Please refer to the documentation :ref:`Documentation <bob.db.verification.filelist>` of this database for more instructions on how to configure this database.
To "plug" your own database in this framework you have to write your own database class by deriving :py:class:`bob.bio.base.database.BioDatabase`.
In this case, you have to derive your class from the :py:class:`bob.bio.base.database.BioDatabase`, and provide the following functions:
In case you want to have a more complicated interface to your database, you are welcome to write your own database wrapper class.
In this case, you have to derive your class from the :py:class:`bob.bio.base.database.Database`, and provide the following functions:
* ``__init__(self, <your-parameters>, **kwargs)``: Constructor of your database interface.
Please call the base class constructor, providing all the required parameters, e.g. by ``bob.bio.base.database.Database.__init__(self, **kwargs)``.
* ``all_files(self)``: Returns a list of all :py:class:`bob.bio.base.database.File` objects of the database.
The list needs to be sorted by the file id (you can use the ``self.sort(files)`` function for sorting).
* ``training_files(self, step, arrange_by_client = False)``: A sorted list of the :py:class:`bob.bio.base.database.File` objects that is used for training.
If ``arrange_by_clients`` is enabled, you might want to use the :py:meth:`bob.bio.base.database.Database.arrange_by_client` function to perform the job.
* ``model_ids(self, group = 'dev'): The ids for the models (usually, there is only one model per client and, thus, you can simply use the client ids) for the given group.
* ``__init__(self, <your-parameters>, **kwargs)`` Constructor of your database interface.
Please call the base class constructor, providing all the required parameters, e.g. by ``super(<your_db>,self).__init__(self, **kwargs)``.
Usually, providing ids for the group ``'dev'`` should be sufficient.
* ``client_id_from_model_id(self, model_id)``: Returns the client id for the given model id.
* ``enroll_files(self, model_id, group='dev')``: Returns the list of model :py:class:`bob.bio.base.database.File` objects for the given model id.
* ``probe_files(self, model_id=None, group='dev')``: Returns the list of probe files, the given model_id should be compared with.
Usually, all probe files are compared with all model files.
In this case, you can just ignore the ``model_id``.
If the ``model_id`` is ``None``, this function is supposed to return *all* probe files for all models of the given group.
* ``objects(self, groups=None, protocol=None, purposes=None, model_ids=None, **kwargs)``
This function must return a list of ``bob.db.base.database.BioFile`` objects with your data.
The keyword arguments are possible filters that you may use.
* ``model_ids_with_protocol(self, groups, protocol, **kwargs)``
This function must return a list of model ids for the given groups and given protocol.
In this context models are basically them "templates" used for enrollment.
Additionally, you can define more lists that can be used for ZT score normalization.
In this case, derive you class from :py:class:`bob.bio.base.database.DatabaseZT` instead, and additionally overwrite the following functions:
If you don't know what ZT score normalization is, just forget about it and move on.
If you know and want to use it, just derive your class from :py:class:`bob.bio.base.database.ZTBioDatabase` instead, and additionally overwrite the following functions:
* ``t_model_ids(self, group = 'dev')``: The ids for the T-Norm models for the given group.
* ``t_enroll_files(self, model_id, group='dev')``: Returns the list of model :py:class:`bob.bio.base.database.File` objects for the given T-Norm model id.
* ``z_probe_files(self, group='dev')``: Returns the list of Z-probe :py:class:`bob.bio.base.database.File` objects, with which all the models and T-Norm models are compared.
* ``tobjects(self, groups=None, protocol=None, model_ids=None, **kwargs)``
This function must return a list of ``bob.db.base.database.BioFile`` objects used for `T` normalization.
* ``zobjects(self, groups=None, protocol=None, **kwargs)``
This function must return a list of ``bob.db.base.database.BioFile`` objects used for `Z` normalization.
* ``tmodel_ids_with_protocol(self, protocol=None, groups=None, **kwargs)``
The ids for the T norm models for the given group and protocol.
.. note:
For a proper biometric recognition protocol, the identities from the models and the T-Norm models, as well as the Z-probes should be different.
For some protocols, a single probe consists of several features, see :ref:`bob.bio.base.algorithms` about strategies how to incorporate several probe files into one score.
If your database should provide this functionality, please overwrite:
* ``uses_probe_file_sets(self)``: Return ``True`` if the current protocol of the database provides multiple files for one probe.
* ``probe_file_sets(self, model_id=None, group='dev')``: Returns a list of lists of :py:class:`bob.bio.base.database.FileSet` objects.
* ``z_probe_file_sets(self, model_id=None, group='dev')``: Returns a list of lists of Z-probe :py:class:`bob.bio.base.database.FileSet` objects (only needed if the base class is :py:class:`bob.bio.base.database.DatabaseZT`).
..
For some protocols, a single probe consists of several features, see :ref:`bob.bio.base.algorithms` about strategies how to incorporate several probe files into one score.
If your database should provide this functionality, please overwrite:
* ``uses_probe_file_sets(self)``: Return ``True`` if the current protocol of the database provides multiple files for one probe.
* ``probe_file_sets(self, model_id=None, group='dev')``: Returns a list of lists of :py:class:`bob.bio.base.database.FileSet` objects.
* ``z_probe_file_sets(self, model_id=None, group='dev')``: Returns a list of lists of Z-probe :py:class:`bob.bio.base.database.FileSet` objects (only needed if the base class is :py:class:`bob.bio.base.database.DatabaseZT`).
.. _bob.bio.base.configuration-files:
......@@ -282,7 +283,7 @@ In ``bob.bio`` this is achieved by providing these parameters in configuration f
In these files, an *instance* of one of the tools is generated, and assigned to a variable with a specific name.
These variable names are:
* ``database`` for an instance of a (derivation of a) :py:class:`bob.bio.base.database.Database`
* ``database`` for an instance of a (derivation of a) :py:class:`bob.bio.base.database.BioDatabase`
* ``preprocessor`` for an instance of a (derivation of a) :py:class:`bob.bio.base.preprocessor.Preprocessor`
* ``extractor`` for an instance of a (derivation of a) :py:class:`bob.bio.base.extractor.Extractor`
* ``algorithm`` for an instance of a (derivation of a) :py:class:`bob.bio.base.algorithm.Algorithm`
......@@ -313,7 +314,7 @@ The process of registering a resource is relatively easy.
We use the SetupTools_ mechanism of registering so-called entry points in the ``setup.py`` file of the according ``bob.bio`` package.
Particularly, we use a specific list of entry points, which are:
* ``bob.bio.database`` to register an instance of a (derivation of a) :py:class:`bob.bio.base.database.Database`
* ``bob.bio.database`` to register an instance of a (derivation of a) :py:class:`bob.bio.base.database.BioDatabase`
* ``bob.bio.preprocessor`` to register an instance of a (derivation of a) :py:class:`bob.bio.base.preprocessor.Preprocessor`
* ``bob.bio.extractor`` to register an instance of a (derivation of a) :py:class:`bob.bio.base.extractor.Extractor`
* ``bob.bio.algorithm`` to register an instance of a (derivation of a) :py:class:`bob.bio.base.algorithm.Algorithm`
......
doc/implemented.rst
+ 2
− 13
View file @ f4c7d6ca
......@@ -14,8 +14,6 @@ Base Classes
bob.bio.base.preprocessor.Preprocessor
bob.bio.base.extractor.Extractor
bob.bio.base.algorithm.Algorithm
bob.bio.base.database.Database
bob.bio.base.database.DatabaseZT
bob.bio.base.grid.Grid
......@@ -30,9 +28,8 @@ Implementations
bob.bio.base.algorithm.LDA
bob.bio.base.algorithm.PLDA
bob.bio.base.algorithm.BIC
bob.bio.base.database.DatabaseBob
bob.bio.base.database.DatabaseBobZT
bob.bio.base.database.DatabaseFileList
bob.bio.base.database.BioDatabase
bob.bio.base.database.ZTBioDatabase
Preprocessors
......@@ -60,7 +57,6 @@ Grid Configuration
.. automodule:: bob.bio.base.grid
.. data:: PREDEFINED_QUEUES
A dictionary of predefined queue keywords, which are adapted to the Idiap_ SGE.
......@@ -68,12 +64,5 @@ Grid Configuration
.. adapted from http://stackoverflow.com/a/29789910/3301902 to ge a nice dictionary content view
.. exec::
import json
from bob.bio.base.grid import PREDEFINED_QUEUES
json_obj = json.dumps(PREDEFINED_QUEUES, sort_keys=True, indent=2)
json_obj = json_obj.replace("\n", "\n ")
print ('.. code-block:: JavaScript\n\n PREDEFINED_QUEUES = %s\n\n' % json_obj)
.. include:: links.rst
doc/installation.rst
+ 24
− 6
View file @ f4c7d6ca
......@@ -17,7 +17,24 @@ Please make sure that you have read the `Dependencies <https://github.com/idiap/
Currently, running Bob_ under MS Windows in not yet supported.
However, we found that running Bob_ in a virtual Unix environment such as the one provided by VirtualBox_ is a good alternative.
The most simple and most convenient way to use the ``bob.bio`` tools is to use a ``zc.buildout`` package, as explained in more detail `here <https://github.com/idiap/bob/wiki/Installation#using-zcbuildout-for-production>`__.
Installation via conda
----------------------
The most simple and most convenient way to use the ``bob.bio`` tools is to install it using `conda <http://conda.pydata.org/docs/intro.html>`_.
We offer pre-compiled binary installations of Bob_ and ``bob.bio`` with conda.
After setting up your conda environment (you can find the instructions `here <https://gitlab.idiap.ch/bob/bob/wikis/Installation>`_), use the sequence of commands to
first search the ``bob.bio`` in the conda-forge channel and then install it.
.. code-block:: sh
$ conda search bob.bio # searching
$ conda install bob.bio.base # Install the base package
Installation via zc.buildout
----------------------------
Another convenient way to use the ``bob.bio`` tools is to use a ``zc.buildout`` package, as explained in more detail `here <https://github.com/idiap/bob/wiki/Installation#using-zcbuildout-for-production>`__.
There, in the ``eggs`` section of the ``buildout.cfg`` file, simply list the ``bob.bio`` packages that you want, like:
.. code-block:: python
......@@ -30,7 +47,7 @@ There, in the ``eggs`` section of the ``buildout.cfg`` file, simply list the ``b
gridtk
in order to download and install all packages that are required for your experiments.
In the example above, you might want to run a video face recognition experiments using the :py:class:`bob.bio.face.preprocessor.FaceDetector` and the :py:class:`bob.bio.face.extractor.DCTBlocks` feature extractor defined in :ref:`bob.bio.face <bob.bio.face>`, the :py:class:`bob.bio.gmm.algorithm.IVector` algorithm defined in :ref:`bob.bio.gmm <bob.bio.gmm>` and the video extensions defined in :ref:`bob.bio.video <bob.bio.video>`, using the YouTube faces database interface defined in :ref:`bob.db.youtube <bob.db.youtube>`.
In the example above, you might want to run a video face recognition experiments using the :py:class:`bob.bio.face.preprocessor.FaceDetect` and the :py:class:`bob.bio.face.extractor.DCTBlocks` feature extractor defined in :ref:`bob.bio.face <bob.bio.face>`, the :py:class:`bob.bio.gmm.algorithm.IVector` algorithm defined in :ref:`bob.bio.gmm <bob.bio.gmm>` and the video extensions defined in :ref:`bob.bio.video <bob.bio.video>`, using the YouTube faces database interface defined in :ref:`bob.db.youtube <bob.db.youtube>`.
Running the simple command line:
.. code-block:: sh
......@@ -42,12 +59,12 @@ will the download and install all dependent packages locally (relative to your c
Databases
~~~~~~~~~
---------
With ``bob.bio`` you will run biometric recognition experiments using some default biometric recognition databases.
Though the verification protocols are implemented in ``bob.bio``, the original data are **not included**.
To download the original data of the databases, please refer to the according Web-pages.
For a list of supported databases including their download URLs, please refer to the :ref:`verification_databases`.
For a list of supported databases including their download URLs, please refer to the `verification_databases <https://github.com/idiap/bob/wiki/Packages>`_.
After downloading the original data for the databases, you will need to tell ``bob.bio``, where these databases can be found.
For this purpose, we have decided to implement a special file, where you can set your directories.
......@@ -69,7 +86,7 @@ Please use ``./bin/databases.py`` for a list of known databases, where you can s
Test your Installation
~~~~~~~~~~~~~~~~~~~~~~
----------------------
One of the scripts that were generated during the bootstrap/buildout step is a test script.
To verify your installation, you should run the script running the nose tests for each of the ``bob.bio`` packages:
......@@ -104,8 +121,9 @@ In case any of the tests fail for unexplainable reasons, please file a bug repor
In other versions, some of the tests may fail.
Generate this documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------
Generally, the documentation of this package is `available online <http://pythonhosted.org/bob.bio.base>`__, and this should be your preferred resource.
However, to generate this documentation locally, you call:
......
requirements.txt
+ 1
− 0
View file @ f4c7d6ca
......@@ -9,5 +9,6 @@ bob.learn.linear
bob.math
bob.measure
bob.sp
gridtk
scipy
setuptools
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment