`memory_demanding` for TF based transformers

There are some situations where you have SampleSets with more than 1000 samples and we can't just TF forward them in one shot without OOM. In this MR I introduce the argument memory_demanding, where, once it's set to true, will `forward one sample at a time.

added 1 commit

• a079dc70 - Add a memory demanding feature in the TF based transformers

Compare with previous version

ping @amohammadi @ydayer @lcolbois

changed title from Add a memory demanding feature in the TF based transformers to {+memory_demanding for+} TF based transformers

changed the description

This is not good @tiago.pereira, you should use dask to split your data.

There are databases that one probe is composed by ~1000 samples. How would you approach that?

So the issue is not because of dask bag size?

sorry I thought this is related to bob.pipelines#27 (closed)

sorry I thought this is related to bob.pipelines#27 (closed)

No, no. The 2 things are not related

I can't break the API just to dask this thing again.

Furthermore, memory_demanding is False by default

why not make this a fixed attribute of the database like the two other attributes here?

Yes, it can be

we need to have a database checker...

The API is getting complicated

I created an attribute checker for the databases, so we don't get confused.

merged

mentioned in commit b385d51c