For a given biometric sample, only one feature can be extracted, and only one identity can be assigned

I will describe the issue in terms of face recognition, but similarities can be drawn to other biometrics, for example speaker recognition.

Currently, we assume that in a given image, there is only one identity in an image. When applying face detection, we only take the face with the highest detection score. In an real open-set have datasets as I have used in my latest opes-set recognition challenge (see: http://vast.uccs.edu/Opensetface) , we have several faces per image, and we need to detect all faces (which might end up in an unknown number of misdetections). Finally, we need to extract a feature vector for each detected bounding box, and compare each feature vector to the gallery models.

So far, none of the concepts that we have implemented thus far is dealing with this kind of problem. I am also not so sure if we should implement a solution directly into bob.bio.base, or if I should try to implement a solution into a stand-alone package.

@amohammadi @andre.anjos @tiago.pereira What do you think?

I think, also the evaluation will be different from our current way, which makes me tending towards a new package. For each image, there must be a list of ground truth identity labels, possibly each bounding box is labeled with an identity.

Now, there are two different ways to evaluate the ground truth. Both of them are open set identification evaluations, in terms of DET plots.

• First, you can just take the identity labels per image and say that a bounding box of a probe is assigned correctly when one of the labeled identities in the image is assigned.
• A bit more difficult, and requiring more information to be carried, the overlap of the detected bounding box and the ground truth bounding box must be considered, and only the identity label of the ground truth bounding box is considered as a correct identification.

Note that I have implemented the latter in our competition evaluation, you can find the script on PyPI: https://pypi.python.org/pypi/challenge.uccs

Hey @mguenther,

I think it fits better in another package. Furthermore, this is another motivation to move forward with this issue (bob.measure#25 (closed)). The four columns structure doesn't fit this task. We would need other meta information (bounding box??).

A bit more difficult, and requiring more information to be carried, the overlap of the detected bounding box and the ground truth bounding box must be considered, and only the identity label of the ground truth bounding box is considered as a correct identification

Is this "overlap" a hyper-parameter of your experiment?

I will look the package that you shared to have more insights.

Cheers

Indeed, the overlap is calculated using the Jaccard index implemented in bob.ip.facedetect.BoundingBox.similarity, better known as Intersection over Union (IOU). Which overlap to threshold at is, indeed, a parameter to the evaluation. A normal value would be 0.5.

It is also true that we would need a different score file structure, which includes the bounding boxes in one way or the other. We have defined one such structure in our competition, see Score File Formats in http://vast.uccs.edu/Opensetface.
If you want to have a look at the (updated) source code for the evaluation, I can send you an invitation to the git repository on bitbucket. The evaluation code on the PyPI page should work as well -- at least on the validation set. I can give you the validation set protocol file if you want to test out the evaluation script itself. Note that we had to play around with the Jaccard index as our ground truth bounding boxes were a bit large.

Thanks for your input. I think I will add this in a different package. First, I'll need to do a proper software design, anyways.

@mguenther a lot of code was not reusable in bob.bio.base to create bob.pad.base on top of it at first. @pkorshunov started slowly refactoring this package to make it more modular. I would expect that you would want to do the same thing.

My suggestion is that do not hesitate to make things more modular and re-usable as possible here. A good approach would even be to have a package like bob.framework or bob.framework.base to put the general technologies there and build bob.bio.base, bob.pad.base, and etc. on top of that.

To have a bob.framework package is, IMHO, not that easily possible. You would need to foresee, which functionality of other packages (which might be developed in the future) should be common between framework packages such as bob.bio and bob.pad. I have designed bob.bio without thinking about other frameworks. It might be very well possible that other frameworks require more or other code that I did not see to be required before.

I think, the package that I am planning to write is more a type of an extension, such as bob.bio.video, that can reuse most parts (face detectors, extractors, algorithms) from bob.bio.face. I don't think (or at least I hope) that I do not need to modify bob.bio.base in order to reach my goal. I will see.

We already have something that goes in this direction (bob.framework). It's called BEAT.

This issue circulates around the definition of a toolchain and the data that flows between the elements of this toolchain. Today this is statically set in bob.bio.base with verify.py defining all the possible toolchain combinations and with FileSelector providing the data to be read. BEAT approaches that by explicitly exposing these elements.

Maybe we should go to this direction, or at least look at the lessons learnt there.

It is not productive spend time to squeeze all recognition problems in bob.bio.base (PAD, Diarization, etc..)

Well, we can start by putting generic code in bob.framework so that it is re-usable by others. Then, we will improve it over time. I would say things like the Grid class, the Singleton Class, and the configuration loading mechanism (both entry points and one wholistic configuration file) are generic enough to be used by other frameworks. Plus many more little things like the is_idiap function, array slicing using SGE variables ... are re-usable.

@tiago.pereira BEAT while clearly having some advantages is not ready as it is. BEAT is planned to change to allow us write frameworks on top of but I do not see that happening any time soon. Even so, in the end it might not be flexible enough for our usage because of security reasons as @andre.anjos pointed out recently (conda environments).

@amohammadi, I'm not suggesting the use of BEAT as is, but at least have a look in the design and see the lessons learnt there.

@tiago.pereira we are planning that. It is supposed to happen over this summer, mainly by @jdiefenbaugh

Also, my TAM which was a week ago was about this.

Really???

Cool, so I'm basically missing everything in the lab :-P

@amohammadi: I think you're confusing 2 different things: (i) what we can do to make beat.core more useful to Bob (ii) what we can do on the Idiap BEAT testbed. The latter requires thinking about security aspects. For the former, we want/need flexibility and debug-ability. We need to figure out a design and tools which allow us to realize (i), but to easily adapt it to (ii) so it can be re-used there.

I agree with @tiago.pereira: making a circle fit on a square will either not be possible or provide poor fit.

I am interested in (i) to be usable without docker and using my own Python environment.

closed