Commit 48b65adf authored by Manuel Günther's avatar Manuel Günther
Browse files

Added functions to retrieve predictions faster.

parent 6bb60b38
""" The module consist of the classes to generate a strong boosting classifier and test features using that classifier.
Boosting algorithms have three main dimensions: weak trainers that are boosting, optimization strategy
for boosting and loss function that guide the optimization. For each one of these the following
Boosting algorithms have three main dimensions: weak trainers that are boosting, optimization strategy
for boosting and loss function that guide the optimization. For each one of these the following
choices are implemented.
Weak Trainers: StumpTrainer- classifies the features based on a specified threshold
LutTrainer- Look-Up-Table are used for classification
Optimization Strategy: For StumpTrainer the gradient descent (GradBoost) is used and for LutTrainer the
optimization is based on Taylor's Boosting framework.
Optimization Strategy: For StumpTrainer the gradient descent (GradBoost) is used and for LutTrainer the
optimization is based on Taylor's Boosting framework.
See following references:
Saberian et. al. "Taylorboost: First and second-order boosting algorithms with explicit
margin control."
......@@ -15,7 +15,7 @@ Optimization Strategy: For StumpTrainer the gradient descent (GradBoost) is used
Loss Function: Exponential Loss (Preferred with the StumpTrainer)
Log Loss (Preferred with LutTrainer)
Tangent Loss
Tangent Loss
"""
......@@ -33,10 +33,10 @@ import scipy.optimize
class Boost:
""" The class to boost the features from a set of training samples.
""" The class to boost the features from a set of training samples.
It iteratively adds new trainer models to assemble a strong classifier.
In each round of iteration a weak trainer is learned
It iteratively adds new trainer models to assemble a strong classifier.
In each round of iteration a weak trainer is learned
by optimization of a differentiable function. The following parameters are involved
......@@ -50,11 +50,11 @@ class Boost:
trainers and Taylor Boost is used as optimization strategy.
Ex.: LBP features, MCT features.
'StumpTrainer': Decision Stumps are used as weak trainer and GradBoost is
'StumpTrainer': Decision Stumps are used as weak trainer and GradBoost is
used as optimization strategy.It can be used with both discrete
and continuous type of features
and continuous type of features
num_rnds: Type int, Default = 100
num_rnds: Type int, Default = 100
The number of rounds for boosting. The boosting strategies implemented here
(GradBoost and TaylorBoost) are fairly robust to overfitting, so the large
number of rounds generally results in a small error rate.
......@@ -62,15 +62,15 @@ class Boost:
loss_type: Type string, Default = 'log'
It is the type of loss function to be optimized. Currently we support the
following classes of loss function:
'log' and 'exp'
'log' and 'exp'
'exp' loss function is preferred with StumpTrainer and 'log' with LutTrainer.
num_entries: Type int, Default = 256
This is the parameter for the LutTrainer. It is the
This is the parameter for the LutTrainer. It is the
number of entries in the LookUp table. It can be determined from the range of
feature values. For examples, for LBP features the number of entries in the
feature values. For examples, for LBP features the number of entries in the
LookUp table is 256.
......@@ -79,8 +79,8 @@ class Boost:
lut_selection: Type string, Default = 'indep'
For multivariate classification during the weak trainer selection the best feature can
either be shared with all the outputs or it can be selected independently for each output.
For feature sharing set the parameter to 'shared' and for independent selection set it to
'indep'. See cosmin's thesis for a detailed explanation on the feature selection type.
For feature sharing set the parameter to 'shared' and for independent selection set it to
'indep'. See cosmin's thesis for a detailed explanation on the feature selection type.
For univariate cases such as face detection this parameter is not relevant.
Example Usage:
......@@ -98,14 +98,14 @@ class Boost:
# Classify the samples using boosted classifier
prediction_labels = machine.classify(test_fea)
"""
def __init__(self, trainer_type, num_rnds = 20, num_entries = 256, loss_type = 'log', lut_selection = 'indep'):
""" The function to initialize the boosting parameters.
""" The function to initialize the boosting parameters.
The function set the default values for the following boosting parameters:
The number of rounds for boosting: 20
......@@ -119,41 +119,41 @@ class Boost:
Values: LutTrainer or StumpTrainer
num_rnds: The number of rounds of boosting
Type: int
Values: 20 (Default)
Values: 20 (Default)
num_entries: The number of entries for the lookup table
Type: int
Values: 256 (Default)
loss_type: The loss function to be be minimized
Type: string
Values: 'log' or 'exp'
Values: 'log' or 'exp'
lut_selection: The selection type for the LUT based trainers
Type: string
Values: 'indep' or 'shared'
Values: 'indep' or 'shared'
"""
self.num_rnds = num_rnds
self.num_entries = num_entries
self.loss_type = loss_type
self.lut_selection = lut_selection
self.weak_trainer_type = trainer_type
def train(self, fset, targets):
""" The function to train a boosting machine.
The function boosts the discrete features (fset) and returns a strong classifier
as a combination of weak classifier.
Inputs:
fset: features extracted from the samples
features should be discrete for lutTrainer.
Type: numpy array (num_sam x num_features)
labels: class labels of the samples
Type: numpy array
The function boosts the discrete features (fset) and returns a strong classifier
as a combination of weak classifier.
Inputs:
fset: features extracted from the samples
features should be discrete for lutTrainer.
Type: numpy array (num_sam x num_features)
labels: class labels of the samples
Type: numpy array
Shape for binary classification: #number of samples
Shape for multivariate classification: #number of samples x #number of outputs
......@@ -169,7 +169,7 @@ class Boost:
"""
# Initializations
# Initializations
if(len(targets.shape) == 1):
targets = targets[:,numpy.newaxis]
......@@ -180,10 +180,10 @@ class Boost:
loss_class = losses.LOSS_FUNCTIONS[self.loss_type]
loss_func = loss_class()
# For lut trainer the features should be integers
# For lut trainer the features should be integers
#if(self.weak_trainer_type == 'LutTrainer'):
# fset = fset.astype(int)
# For each round of boosting initialize a new weak trainer
if self.weak_trainer_type == 'LutTrainer':
......@@ -197,7 +197,7 @@ class Boost:
# Start boosting iterations for num_rnds rounds
for r in range(self.num_rnds):
# Compute the gradient of the loss function, l'(y,f(x)) using loss_class
loss_grad = loss_func.update_loss_grad(targets,pred_scores)
......@@ -206,24 +206,24 @@ class Boost:
# Compute the classification scores of the samples based only on the current round weak classifier (g_r)
curr_pred_scores = curr_weak_trainer.get_weak_scores(fset)
# Initialize the start point for lbfgs minimization
init_point = numpy.zeros(num_op)
# Perform lbfgs minimization and compute the scale (alpha_r) for current weak trainer
lbfgs_struct = scipy.optimize.fmin_l_bfgs_b(loss_func.loss_sum, init_point, fprime = loss_func.loss_grad_sum, args = (targets, pred_scores, curr_pred_scores))
lbfgs_struct = scipy.optimize.fmin_l_bfgs_b(loss_func.loss_sum, init_point, fprime = loss_func.loss_grad_sum, args = (targets, pred_scores, curr_pred_scores))
alpha = lbfgs_struct[0]
# Update the prediction score after adding the score from the current weak classifier f(x) = f(x) + alpha_r*g_r
pred_scores = pred_scores + alpha*curr_pred_scores
pred_scores = pred_scores + alpha*curr_pred_scores
# Add the current trainer into the boosting machine
machine.add_weak_trainer(curr_weak_trainer, alpha)
return machine
......@@ -236,48 +236,57 @@ class Boost:
class BoostMachine():
""" The class to perform the classification using the set of weak trainer """
def __init__(self, num_op):
""" Initialize the set of weak trainers and the alpha values (scale)"""
self.alpha = []
self.weak_trainer = []
self.num_op = num_op
def add_weak_trainer(self, curr_trainer, curr_alpha):
""" Function adds a weak trainer and the scale into the list
Input:
Input:
curr_trainer: the weak trainer learner during a single round of boosting
curr_alpha: the scale for the curr_trainer
"""
self.alpha.append(curr_alpha)
self.weak_trainer.append(curr_trainer)
def __call__(self, feature):
"""Returns the predicted score for the given single feature, assuming only single output.
Input: A single feature vector of length No. of total features
Output: A single floating point number
"""
return sum([a * weak.get_weak_score(feature) for (a, weak) in zip(self.alpha, self.weak_trainer)])
def classify(self, test_features):
""" Function to classify the test features using a strong trained classifier.
The function classifies the test features using the boosting machine trained with a
The function classifies the test features using the boosting machine trained with a
combination of weak classifiers.
Inputs:
test_features: The test features to be classified using the trained machine
Type: numpy array (#number of test samples x #number of features)
Return:
Return:
prediction_scores: The real valued number which are thresholded to determine the prediction classes.
prediction_labels: The predicted classes for the test samples. It is a binary numpy array where
prediction_labels: The predicted classes for the test samples. It is a binary numpy array where
1 indicates the predicted class.
Type: numpy array
Type: numpy array
Shape for binary classification: #number of samples
Shape for multivariate classification: #number of samples x #number of outputs
Examples for 4 classes case (0,1,2,3) and three test samples.
......@@ -285,13 +294,13 @@ class BoostMachine():
[ 1, -1, -1, -1], #Predicted class is 0
[-1, -1, -1, 1]] #Predicted class is 3
There can be only single 1 in a row and the index of 1 indicates the class.
"""
# Initialization
num_trainer = len(self.weak_trainer)
num_samp = test_features.shape[0]
pred_labels = -numpy.ones([num_samp, self.num_op])
pred_scores = numpy.zeros([num_samp, self.num_op])
pred_scores = numpy.zeros([num_samp, self.num_op])
# For each round of boosting calculate the weak scores for that round and add to the total
......
""" The module consists of the weak trainers which are used in the boosting framework.
currently two trainer types are implemented: Stump trainer and Lut trainer.
The modules structure is as follows:
currently two trainer types are implemented: Stump trainer and Lut trainer.
The modules structure is as follows:
StumpTrainer class provides the methods to compute the weak stump trainer
and test the features using these trainers.
and test the features using these trainers.
LutTrainer class provides the methods to compute the weak LUT trainer
and test the features using these trainers.
and test the features using these trainers.
"""
......@@ -28,20 +28,20 @@ class StumpMachine():
def get_weak_scores(self,test_features):
""" The function to perform classification using a weak stump classifier.
The function computes the classification scores for the test features using
a weak stump trainer. Since we use the stump classifier the classification
The function computes the classification scores for the test features using
a weak stump trainer. Since we use the stump classifier the classification
scores are either +1 or -1.
Input: self: a weak stump trainer
test_features: A matrix of the test features of dimension.
test_features: A matrix of the test features of dimension.
Num. of Test images x Num. of features
Return: weak_scores: classification scores of the test features use the weak classifier self
Array of dimension = Num. of samples
Array of dimension = Num. of samples
"""
# Initialize the values
numSamp = test_features.shape[0]
weak_scores = numpy.ones([numSamp,1])
# Select feature corresponding to the specific index
weak_features = test_features[:,self.selected_indices]
......@@ -52,30 +52,30 @@ class StumpMachine():
class StumpTrainer():
""" The weak trainer class for training stumps as classifiers. The trainer is parametrized
the threshold and the polarity.
""" The weak trainer class for training stumps as classifiers. The trainer is parametrized
the threshold and the polarity.
"""
def compute_weak_trainer(self, fea, loss_grad):
""" The function to compute weak Stump trainer.
""" The function to compute weak Stump trainer.
The function computes the weak stump trainer. It is called at each boosting round.
The best weak stump trainer is chosen to maximize the dot product of the outputs
The best weak stump trainer is chosen to maximize the dot product of the outputs
and the weights (gain). The weights in the Adaboost are the negative of the loss gradient
for exponential loss.
Inputs:
Inputs:
fea: the training feature set
loss_grad: the gradient of the loss function for the training samples
Chose preferable exponential loss function to simulate Adaboost
Chose preferable exponential loss function to simulate Adaboost
Return:
Return:
self: a StumpTrainer Object, i.e. the optimal trainer that minimizes the loss
"""
......@@ -103,9 +103,9 @@ class StumpTrainer():
def compute_thresh(self, fea ,loss_grad):
""" Function computes the stump classifier (threshold) for a single feature
Function to compute the threshold for a single feature. The threshold is computed for
""" Function computes the stump classifier (threshold) for a single feature
Function to compute the threshold for a single feature. The threshold is computed for
the given feature values using the weak learner algorithm of Viola Jones.
Inputs:
......@@ -126,7 +126,7 @@ class StumpTrainer():
# Sort the feature and rearrange the corresponding weights and feature values
sorted_id = numpy.argsort(fea)
fea = fea[sorted_id]
fea = fea[sorted_id]
loss_grad = loss_grad[sorted_id]
# For all the threshold compute the dot product
......@@ -135,7 +135,7 @@ class StumpTrainer():
gain = (grad_sum - grad_cs)
# Find the index that maximizes the dot product
opt_id = numpy.argmax(numpy.absolute(gain))
opt_id = numpy.argmax(numpy.absolute(gain))
gain_max = numpy.absolute(gain[opt_id])
# Find the corresponding threshold value
......@@ -160,20 +160,20 @@ class StumpTrainer():
class LutMachine():
""" The LUT machine consist of the core elements of the LUT weak classfier i.e. the LUT and
""" The LUT machine consist of the core elements of the LUT weak classfier i.e. the LUT and
the feature index corresponding to the weak classifier. """
def __init__(self, num_outputs, num_entries):
""" The function initializes the weak LUT machine.
The function initializes the look-up-table and the feature indices of the LUT machine.
Inputs:
Inputs:
self:
num_entries: The number of entries for the LUT
type: int
num_outputs: The number of outputs for the classification task.
num_outputs: The number of outputs for the classification task.
type: Integer
"""
......@@ -183,14 +183,14 @@ class LutMachine():
def get_weak_scores(self, features):
""" Function computes classification results according to the LUT machine
""" Function computes classification results according to the LUT machine
Function classifies the features based on a single LUT machine.
Function classifies the features based on a single LUT machine.
Input:
Input:
fset: The set test features. No. of test samples x No. of total features
return:
return:
weak_scores: The classification scores of the features based on current weak classifier"""
# Initialize
......@@ -204,51 +204,62 @@ class LutMachine():
return weak_scores
def get_weak_score(self, feature):
"""Returns the weak score for the given single feature, assuming only a single output.
Input: a single feature vector of size No. of total features.
Output: a single number (+1/-1)
"""
return self.luts[feature[self.selected_indices[0]],0]
class LutTrainer():
""" The LutTrainer class contain methods to learn weak trainer using LookUp Tables.
""" The LutTrainer class contain methods to learn weak trainer using LookUp Tables.
It can be used for multi-variate binary classification """
def __init__(self, num_entries, selection_type, num_outputs):
""" Function to initialize the parameters.
Function to initialize the weak LutTrainer. Each weak Luttrainer is specified with a
LookUp Table and the feature index which corresponds to the feature on which the
current classifier has to applied.
Function to initialize the weak LutTrainer. Each weak Luttrainer is specified with a
LookUp Table and the feature index which corresponds to the feature on which the
current classifier has to applied.
Inputs:
Inputs:
self:
num_entries: The number of entries for the LUT
type: int
selection_type: The feature selection can be either independent or shared. For independent
selection_type: The feature selection can be either independent or shared. For independent
case the loss function is separately considered for each of the output. For
shared selection type the sum of the loss function is taken over the outputs
and a single feature is used for all the outputs. See Cosmin's thesis for more details.
Type: string {'indep', 'shared'}
num_outputs: The number of outputs for the classification task.
num_outputs: The number of outputs for the classification task.
type: Integer
"""
self.num_entries = num_entries
self.num_outputs = num_outputs
self.selection_type = selection_type
def compute_weak_trainer(self, fea, loss_grad):
""" The function to learn the weak LutTrainer.
The function searches for a features index that minimizes the the sum of the loss gradient and computes
the LUT corresponding to that feature index.
""" The function to learn the weak LutTrainer.
The function searches for a features index that minimizes the the sum of the loss gradient and computes
the LUT corresponding to that feature index.
Inputs:
self: empty trainer object to be trained
fea: The training features samples
type: integer numpy array (#number of samples x number of features)
......@@ -264,7 +275,7 @@ class LutTrainer():
fea_grad = numpy.zeros([self.num_entries, self.num_outputs])
lut_machine = LutMachine(self.num_outputs, self.num_entries)
# Compute the sum of the gradient based on the feature values or the loss associated with each
# Compute the sum of the gradient based on the feature values or the loss associated with each
# feature index
sum_loss = self.compute_grad_sum(loss_grad, fea)
......@@ -272,7 +283,7 @@ class LutTrainer():
# Select the most discriminative index (or indices) for classification which minimizes the loss
# and compute the sum of gradient for that index
if self.selection_type == 'indep':
# indep (independent) feature selection is used if all the dimension of output use different feature
......@@ -288,7 +299,7 @@ class LutTrainer():
elif self.selection_type == 'shared':
# for 'shared' feature selection the loss function is summed over multiple dimensions and
# for 'shared' feature selection the loss function is summed over multiple dimensions and
# the feature that minimized this cumulative loss is used for all the outputs
accum_loss = numpy.sum(sum_loss,1)
......@@ -298,40 +309,40 @@ class LutTrainer():
for output_index in range(self.num_outputs):
fea_grad[:,output_index] = self.compute_grad_hist(loss_grad[:,output_index],fea[:,selected_findex])
# Assign the values to LookUp Table
lut_machine.luts[fea_grad <= 0.0] = -1
return lut_machine
def compute_grad_sum(self, loss_grad, fea):
""" The function to compute the loss gradient for all the features.
The function computes the loss for whole set of features. The loss refers to the sum of the loss gradient
of the features which have the same values.
Inputs:
loss_grad: The loss gradient for the features. No. of samples x No. of outputs
Inputs:
loss_grad: The loss gradient for the features. No. of samples x No. of outputs
Type: float numpy array
fea: set of features. No. of samples x No. of features
Output:
Output:
sum_loss: the loss values for all features. No. of samples x No. of outputs"""
# initialize values
num_fea = len(fea[0])
num_samp = len(fea)
sum_loss = numpy.zeros([num_fea,self.num_outputs])
# Compute the loss for each feature
for feature_index in range(num_fea):
for output_index in range(self.num_outputs):
hist_grad = self.compute_grad_hist(loss_grad[:,output_index],fea[:,feature_index])
sum_loss[feature_index,output_index] = - sum(abs(hist_grad))
return sum_loss
......@@ -342,9 +353,9 @@ class LutTrainer():
def compute_grad_hist(self, loss_grado,features):
""" The function computes the loss for a single feature.
Function computes sum of the loss gradient that have same feature values.
Function computes sum of the loss gradient that have same feature values.
Input: loss_grado: loss gradient for a single output values. No of Samples x 1
fval: single feature selected for all samples. No. of samples x 1
......@@ -352,7 +363,7 @@ class LutTrainer():
# initialize the values
# hist_grad = numpy.zeros([self.num_entries])
# compute the sum of the gradient
hist_grad, bin_val = numpy.histogram(features, bins = self.num_entries, range = (0,self.num_entries-1), weights = loss_grado)
......@@ -375,21 +386,21 @@ class GaussianMachine():
num_classes = self.means.shape[0]
num_features = features.shape[0]
scores = numpy.zeros([num_features,num_classes])
for i in range(num_classes):
mean_i = self.means[i]