Issue 8 remove database configuration

This merge request sediments what we have discussed along the week. Basically I removed bob.bio.db package and move the BioDatabase and BioFile classes to bob.bio.base. I will branch the bob.nightlies to better do the integration tests.

I will leave this merge request open, just to not forget

Why did you revert my changes here?

My problem was (is) that bob.db.atnt does not return bob.bio.db.File objects, but bob.db.base.File objects. However, most probably a cleaner solution would be to have a DummyFile inside here, which derives from bob.bio.db.File and return this throughout, i.e., mimicking the other databases. I will check on that.

Yes please do that. You don't need to create a DummyFile, you can convert the files on the fly to bob.bio.db.File.

I see you have done it already, thanks.

Hey @mguenther , we also need to merge this branch (https://gitlab.idiap.ch/bob/bob.bio.base/tree/29-preprocessor-does-not-use-the-load-method-of-the-biofile-class) right ?

Yes, indeed, this is part of this branch. Please note that there needs to be some action taken in derived bob.bio classes, where the preprocessors need to be adapted to the new interface. We can open a new branch for that, or just do it inside of this branch -- as the preprocessor and database are somewhat related.

Added 19 commits:

• ed59a2f8 - Revert "Fix for bob.bio.video"
• d69bcf71 - Revert "use the File.load method if possible."
• a7a9a0d3 - First version of proposed handling of original data
• 4373e7e0 - Updated dummy databases to work with bob.db.atnt directly and return BioFile or BioFileSet objects
• 63d3b73e - Updated name of load_function to be read_original_data (to be consistent with th…
• cc3e85e6 - Using derived class BioFile.load functions as default for read_original_data
• 43a4efcb - Introduced the --configuration-file command line option
• ffcb9385 - Add more cmdline options
• de870e6c - Fix argument parsing destination
• 988a7ca3 - Make import relative
• 8c3b1190 - Fixes and tests for the config-file feature
• 25b5d50f - Fix py3 compatibility issue
• 9f5f7e6d - Better fix w/o leaving something untested
• 9bc7a152 - Use string mode to open tempfile for py3 compat
• b26bf656 - [refactoring2016] Merged 29-preprocessor-does-not-use-the-load-method-of-the-bio…
• 104010ae - [refactoring2016] Moved bob.bio.db.BioDatabase to bob.bio.base
• ce5b6498 - [refactoring2016] Updated documentation and buildout
• 432b2302 - [refactoring2016] Updated the annotations function
• 451ca636 - Merge branch 'issue-8-remove-database-configuration' of gitlab.idiap.ch:bob/bob.…

Just rebased this branch with 29-preprocessor-does-not-use-the-load-method-of-the-biofile-class , fixed some issues and I'm running some experiments.

Everything is green so far https://gitlab.idiap.ch/bob/bob.bio.base/pipelines

I've tested bob.bio.face and bob.bio.spear. Everything is working fine, but I need an extra pair of eyes :-)

If you have some time to spend on this, would be great @bob

Sure, I’ll try it this week.

After updating the master branches of the packages you told me, train_gmm.py in bob.bio.spear stopped working yesterday for voxforge, while it used to work before… (train_gmm.py asks for the algorithm to be specified, even when it is specified and has a ‘gmm’ defalut , I think it might be related to the configuration file option you introduced) :( I’ll try again tomorrow after all these updates.

Thanks!

This is great work, Tiago - @mguenther, @amohammadi: do you agree we go ahead and merge this?

bob.bio.video returns a list of file_objects. Calling list.make_path will not work.

Do we really need file_name here? except for logging.

Maybe we can log in the Preprocessor class?

bob.bio.video should return a BioVideoFile object, not a list of BioFile objects. This should be part of the refactoring.

Indeed, file_name is only used for logging. However, I would like to keep this here, so that not every preprocessor needs to deal with logging.

So how do we deal with this? How can we have bob.db.atnt and bob.db.mobio working with bob.bio.video again?

I think we can create BioVideoFile to be something like this:

class VideoBioFile(BioFile):

    def __init__(self, client_id, path, file_id, paths=None):
        """
        `paths` can be a list of images If your database contains a list of images instead of a video file otherwise you don't need to provide it.
        """
        super(VideoBioFile, self).__init__(client_id=client_id, path=path, file_id=file_id)
        self.paths = paths

    def make_path(self, directory, extension):
        '''should either return a path to a video file or a list of path to images'''
        if self.paths:
            raise NotImplementedError('Please override this method if your databases contains a list of images instead of a video')
        else:
            return super(VideoBioFile, self).make_path(directory, extension)

    def load(self, directory, extension):
        # bob.io.base.load can handle a list of images to load or a video file to load?
        data = bob.io.base.load(self.make_path(directory, extension))
        data = numpy.array(data) # make data a 4 dimension numpy array [frames, y, x, color] depends on what FrameContainer expects.
        return FrameContainer(data)

Hi @tiago.pereira I see that you canceled the tests for this on nightlies. Can we have this and other packages with these changes to run on nightlies please?

Yes, sure. I'm debugging bob.bio.video now. When finish I will go back to this

I think I handled the bob.bio.video in an elegant way. Let's wait the builds (it is working on my computer :-P ) https://gitlab.idiap.ch/bob/bob.bio.video/builds/11833.

So, the default behaviour is like this: https://gitlab.idiap.ch/bob/bob.bio.video/blob/issue-8-remove-database-configuration/bob/bio/video/database/database.py#L24

    def load(self, directory=None, extension='.avi'):
        return FrameSelector()(self.make_path(directory, extension))

The dummy video database: https://gitlab.idiap.ch/bob/bob.bio.video/blob/issue-8-remove-database-configuration/bob/bio/video/test/dummy/database.py#L10

class DummyBioFile(VideoBioFile):
    def load(self, directory=None, extension='.avi'):
        file_name = self.make_path(directory, extension)
        fc = FrameContainer()
        fc.add(os.path.basename(file_name), bob.io.base.load(file_name))
        return fc

And finally the Youtube (which opens a list of files): https://gitlab.idiap.ch/bob/bob.bio.video/blob/issue-8-remove-database-configuration/bob/bio/video/database/youtube.py#L20

class YoutubeBioFile(VideoBioFile):
    def load(self, directory=None, extension='.jpg'):
        files = os.listdir(self.make_path(directory, ''))
        fc = FrameContainer()

        for f in files:
            if extension == os.path.splitext(f)[1]:
                file_name = os.path.join(self.make_path(directory, ''), f)
                fc.add(os.path.basename(file_name), bob.io.base.load(file_name))
        return fc

Added 19 commits:

• 451ca636...28a640f6 - 7 commits from branch master
• 753d3f93 - Revert "Fix for bob.bio.video"
• 10f2493c - Revert "use the File.load method if possible."
• 1d88685f - First version of proposed handling of original data
• 04ff36c2 - Updated dummy databases to work with bob.db.atnt directly and return BioFile or BioFileSet objects
• fdd1a334 - Updated name of load_function to be read_original_data (to be consistent with th…
• 3d6d2148 - Using derived class BioFile.load functions as default for read_original_data
• 67916c31 - [refactoring2016] Moved bob.bio.db.BioDatabase to bob.bio.base
• 3030866d - [refactoring2016] Updated documentation and buildout
• f8a50678 - [refactoring2016] Updated the annotations function
• ed0617d8 - [refactoring2016] Merged 29-preprocessor-does-not-use-the-load-method-of-the-bio…
• 0ef1c256 - [refactoring2016] Moved bob.bio.db.BioDatabase to bob.bio.base
• fafbda49 - Merge branch 'issue-8-remove-database-configuration' of gitlab.idiap.ch:bob/bob.…

I will merge this one

mentioned in commit 554419d6

Status changed to merged

mentioned in merge request bob.bio.spear!14 (merged)

mentioned in commit bob.bio.spear@8352cb41

mentioned in merge request bob.bio.face!6 (merged)

mentioned in merge request bob.bio.vein!5 (merged)