Fix protocols with the same name ending being mixed up

CSVDataset protocols with the same ending in their name (e.g. female and male) are no longer mixed up.

added Easy label

assigned to @ydayer

requested review from @amohammadi

Related to bob.extension#84 (closed).

merged

mentioned in commit de647952

This broke bob.bio.face

from bob.extension.download import search_file
dataset_protocol_path = "/idiap/temp/tpereira/bob/datasets/mobio.tar.gz"
protocol = "mobile0-male-female"
f = search_file(dataset_protocol_path,[os.path.join("/", protocol, "norm", "train_world.lst"),os.path.join("/", protocol, "norm", "train_world.csv"),],)
print(f is None)
.. True

Hey @ydayer, which package/database did you test this ?

Thanks

:( I tested with the csv files I created for bob.bio.spear (this one, for example).

The only difference I see, is the structure of the tar, the one that works is /./protocol/... and the one that don't is /protocol/...

mmmm,

So, are you suggesting that we should have in our tar-balls this structure ./database-name/protocol/dev..... instead of ./protocol/dev..... ?

Yes, shit, multipie works and has this structure multipie/P240/norm/train_world.csv for example. and mobio has laptop1-female/eval/for_models.csv.

The matching strategy is not ideal yet.

The other solution would be to fully match the string when searching, instead of what is done now (if str in path_name)

Btw, I created those archives with this structure by following the example in the wiki (here: tar.add(output, arcname="."))

Yes, let's match the fullname then

Well, matching a full-name will be a problem too

Yes but more manageable than re-creating every problematic dataset archive, I guess.

Matching fullname is worst, because we would have no choice rather than changing the structure of the tarballs.

Well, I think we have no choice rather than changing the tarballs and changing the search mechanism to the following pattern

search_file(dataset_protocol_path,[os.path.join(name, protocol, "norm", "train_world.lst")

In this way we have no holes in the interface.

What do you think?

The more I think of it the worse it gets. I was opting to strip the eventual /./ in front to then do a full match. But turns out that tarfile renames /./ into mobio/ or whatever the db name is:

from bob.extension.download import search_file
import os
import tarfile
dataset_protocol_path = "/idiap/temp/ydayer/bob_data/datasets/mobio.tar.gz"
protocol = "mobile0-male-female"
with tarfile.open(dataset_protocol_path) as f:
    for member in f.getmembers():
        if member.isdir():
            continue
        print(member.name)

prints

mobio/mobile0-male-female/dev/for_models.csv
mobio/mobile0-male-female/dev/for_probes.csv
mobio/mobile0-male-female/eval/for_models.csv
mobio/mobile0-male-female/eval/for_probes.csv
mobio/mobile0-male-female/norm/for_tnorm.csv
[...]

Well yes, adding name in the search and standardizing the structure would be the correct solution. Do you have an idea of how many dbs would need to be re-archived?

My tarball is different than yours.

$ md5sum /idiap/temp/ydayer/bob_data/datasets/mobio.tar.gz
42cee778c17a34762d5fc5dd13ce3ee6  /idiap/temp/ydayer/bob_data/datasets/mobio.tar.gz

$ md5sum /idiap/temp/tpereira/bob/datasets/mobio.tar.gz
eb55f3fda07c6108888a6bd711be66d5  /idiap/temp/tpereira/bob/datasets/mobio.tar.gz

I think you don't have the most updated bob.bio.face

well, I'll standardize everything

thanks man.

My tarball is different than yours

Sorry about that, I have a mobio-audio dataset that I created recently, not mobio-image, hence the difference.

Well yes, adding name in the search and standardizing the structure would be the correct solution. Do you have an idea of how many dbs would need to be re-archived?

Several aahhaha, I'm on it. I was fixing something, and I ended up fixing something else ahahhaha

@ydayer, I'll merge this one, where I standardize the CSV interface file structure.

!280 (merged)

I think this will not impact your tests, but, please, run your tests just in case.

Cheers