New processor API

Working with the new bob.pipelines API in bob.bio.base.

added 2 commits

• 9324175d - Reverted to legacy code
• 8dbf5844 - Base code for the comparator

Compare with previous version

changed title from WIP: Created Processors for linearized and PCA to WIP: New processor API

mentioned in merge request bob.pipelines!6 (merged)

added 1 commit

• 62ddf1a3 - Integration bob.bio.base

Compare with previous version

Hi guys, this is one step towards the integration of our new pipeline mechanism with bob.bio.base. If you buildout this branch developing bob.pipelines, you can type this example to see how it goes.

./bin/bob pipelines vanilla-biometrics ./src/bob.bio.base/bob/bio/base/config/baselines/pca_atnt.py  -o ./example/

Following the discussions with @amohammadi, follow the skeleton of the vanilla-biometrics pipeline. https://gitlab.idiap.ch/bob/bob.bio.base/blob/new-processing-API/bob/bio/base/pipelines/vanilla_biometrics/pipeline.py

It didn't changed much, but basically all the checkpoint shenenigans are gone. Comparators and Estimators should handle this themselves. WARNING. This is not the final setup, but it's going to look like more/less like this,

Follow below a macro list of tasks to be completed before the final merge (not to master, but to the other branch)

• Make this pipeline work with dask. The idea is; once you do dask_it(pipeline) everything should work flawlessly. Propably we would need to play around with https://gitlab.idiap.ch/bob/bob.bio.base/blob/62ddf1a3bc440560044413d15c4d904569c5ac46/bob/bio/base/pipelines/vanilla_biometrics/pipeline.py#L68 to make it work properly. Let's see.
• Implement the checkpoint mixing for the Comparators.
• Implement Mixins for the legacy stuff
• Implement Comparator for the legacy stuff
• Remove unecessary stuff from the package (file database API for instance)

added 2 commits

• f12f4e8c - Checkpointing enroll
• 90dc9a6b - Implemented checkpoint mechanism

Compare with previous version

Our Comparators (rebranded it to BiometricAlgorithm) now supports checkpointing in the same fashion as pipelines.

Don't know how I'm going to "dask_it", but this is a problem for tomorrow.

Cheers

added 1 commit

• 734eb826 - Fixed issue in the Checkpoint scoring

Compare with previous version

added 1 commit

• d754d3cf - Dasking the vanilla pipeline

Compare with previous version

added 1 commit

• 328452af - Patched config file

Compare with previous version

Hi guys,

Another step foward in the vanilla-biometrics. Still have some work to be done (look at the macro checklist above).

Today I put in place the dask thingy.

Now, if you want to dask the WHOLE vanilla-biometrics, you just need to dask_it the scikit pipeline. If want to debug and run stuff locally, just remove the dask verb that everything works flawlessly. You can also checkpoint or not your things

Follow a possible config file to be triggered with bob pipelines vanilla-biometrics.

# Database setup
import bob.db.atnt
from bob.bio.base.pipelines.vanilla_biometrics.legacy import DatabaseConnector
database = DatabaseConnector(bob.db.atnt.Database(), protocol="Default")

from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.decomposition import PCA

# USING THE PCA FROM SCIKIT MIXED WITH OUR STUFF
from bob.pipelines.mixins import CheckpointMixin, SampleMixin
from bob.bio.base.mixins import CheckpointSampleLinearize
class CheckpointSamplePCA(CheckpointMixin, SampleMixin, PCA):   pass

from bob.pipelines.mixins import dask_it
extractor = Pipeline(steps=[('0',CheckpointSampleLinearize(features_dir="./example/extractor0")), 
	                        ('1',CheckpointSamplePCA(features_dir="./example/extractor1", model_path="./example/pca.pkl"))])

# DASKING THE PIPELINE
extractor = dask_it(extractor)

# BIOMETRIC ALGORITHM
from bob.bio.base.pipelines.vanilla_biometrics.biometric_algorithm import Distance, BiometricAlgorithmCheckpointMixin
class CheckpointDistance(BiometricAlgorithmCheckpointMixin, Distance):  pass
algorithm = CheckpointDistance(features_dir="./example/")

Let me know what do you think.

Cheers

added 1 commit

• 499ed374 - Created legacy mixin for preprocessor and extractors

Compare with previous version

added 1 commit

• dc22f7ea - Handling Legacy

Compare with previous version

Hi guys,

Update.

1. I implemented something on bob.pipelines that helps us to integrate our Samples with scikit (and ours) supervised models (https://gitlab.idiap.ch/bob/bob.bio.base/blob/dc22f7ea72ce4656ebac7b925ff43f88fcb823eb/bob/bio/base/pipelines/vanilla_biometrics/pipeline.py#L44).

2. The legacy integration is half way done. Full update when it's 100% done.

added 1 commit

• 6c2449ed - Finished legacy Mixins

Compare with previous version

added 1 commit

• debfe7de - Finished legacy comparator

Compare with previous version

BiometricAlgorithm (a.k.a Comparator) is finished for legacy stuff.

Basically i created 3 classes to handle our legacy stuff on bob.bio.base.

1. LegacyProcessorMixin (https://gitlab.idiap.ch/bob/bob.bio.base/blob/debfe7de112f4195593f63aad2abec5117deeadd/bob/bio/base/mixins/legacy.py#L43). This one is supposed to handle Preprocessors and Extractors. They can be mixed in with whaterver mixins we created. Basically its transform function maps with __call__

2. LegacyAlgorithmMixin (https://gitlab.idiap.ch/bob/bob.bio.base/blob/debfe7de112f4195593f63aad2abec5117deeadd/bob/bio/base/mixins/legacy.py#L91). This one handles one part of bob.bio.base.algorithm.Algorithm. Its fit maps to the bob.bio.base.algorithm.Algorithm.train_projector and its transform maps to bob.bio.base.algorithm.Algorithm.project. Since the train_projector function always save its content, this class HAS TO BE CHECKPOINTABLE.

3. LegacyBiometricAlgorithm (https://gitlab.idiap.ch/bob/bob.bio.base/blob/debfe7de112f4195593f63aad2abec5117deeadd/bob/bio/base/pipelines/vanilla_biometrics/legacy.py#L195). This one handles the second part of the bob.bio.base.algorithm.Algorithm, which is do the enroll`score`.

Monday I'll do a major clean up in this MR and I would like to meet with you either on Tuesday/Wednesday for a personal (zoom) discussion.

Cheers and have a nice weekend

added 2 commits

• 87e74dbe - Cleaning up
• b9f0975b - [black]fy

Compare with previous version

added 2 commits

• 90577c21 - Added some deprecation warnings
• d4c01871 - Removed openBR scoring

Compare with previous version

added 2 commits

• 5e89c744 - Removed openBR scoring
• 93fac719 - Deprecation warnings

Compare with previous version