Updated legacy checkpoint wrapper to used pickle+gzip format

Looks to help, it improved the speed by a bit on SCFace (although it's still quite slower than in non legacy pipelines). Now compute_scores and write_scores seem to take around the same time which is already quite better.

I'm running a FRGC test to see how it goes.

Looks pretty slow still, on the FRGC test I didn't manage yet to finish any write_scores task. Currently write_scores took 60% of the execution time (33% for the compute_scores), and growing.

Of the write_scores task, we spend most of it in the uncompress_unpickle function, which I guess makes sense. But still it's weird that it's so slow as I don't remember it ever being an issue before. I am wondering if it's linked to one of my library versions, my env does not have bob-devel in it so I might have some unpinned stuff that is playing tricks.

Let me try on my side. Which protocol are you running?

FRGC exp1 :

./bin/bob bio pipelines vanilla-biometrics frgc-exp1 gabor_graph -m --dask-partition-size 50 -c -vv -l sge

Can we merge this?

Yes we can. The FRGC issues are still remaining but are unrelated to this improvement which makes sense by itself.

merged

mentioned in commit e6b0ccd0

mentioned in issue bob#269 (closed)