Tests takes around 3 hours to finish and use more than 16GB of memory

@tiago.pereira these new tests are really stopping me to do my work. I am testing Bob packages on a new machine with 16 GB of RAM, and it crashes my machine. Could you please make these unit tests more realistic?

changed milestone to %Bob 9.0.0

Hey man, what do you mean by realistic?

I'm testing the output of all CNN models we have. We have several of them (I'll push more in the following weeks). This is realistic.

are you doing batch inference? that takes more RAM than needed. Why does the tests take 1h to finish?

Yes, I'm doing batch inference. What takes time is the download and model loading.

What can be done to reduce this time is to reduce the amount of tests with these CNN. Today we test the:

• Extractors (https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/master/bob/bio/face/test/test_embeddings.py)
• Resource loading (https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/master/bob/bio/face/test/test_transformers.py)
• Baselines: (https://gitlab.idiap.ch/bob/bob.bio.face/-/blob/master/bob/bio/face/test/test_baselines.py)

All of them are important, but we can reduce to test only the baselines.

by realistic I mean the tests should not take more than 8GB of RAM and should finish in less than 30 minutes.

I don't see this as a problem. I'm testing all the cases these models can be loaded.

We can either reduce the coverage of these tests or decorate them with a flag so it runs once a week on the CI.

why do you think using more than 16GB of RAM for unit tests is not a problem?

I've never checked the memory throughput of these tests. The tests are rather simple. Might be something related with TF or MXNet. I don't know. Needs to be investigated (worth to investigate? the CI can handle that)

What I can tell for sure is that my experiments using these models are far from using 16GB.

Let's start by marking these tests as slow. I can tell you that in my other machine where I have 32 GB of RAM the tests are using 28 GB of RAM.

mentioned in commit 3dd816da

mentioned in merge request !107 (merged)

I did not close this because I would like to see the RAM usage to be below 16 GB RAM.

This is tensorflow @amohammadi

We've spotted some memory leak with keras.predict in TF <= 2.4

I'm using TF nightly (TF 2.5.x) and these issues seems to be vanished for me.

This issue tends to be opened for ages. Furthermore, our CI can handle this. Would be better to close it

FYI, since !112 (merged) has been merged, the tests are now taking 3 hours to finish!

changed title from Tests takes around 1 hour to finish and use more than 16GB of memory to Tests takes around 3 hours to finish and use more than 16GB of memory

taking ~45 minutes now

I think the only change that we did that tests went from 5 hours to 45 minutes was this bob.pipelines!69 (merged)

closed

@tiago.pereira I cannot test bob.bio.face on my computer. I only have 16GB of RAM and tests fail (out of memory crash) at:

src/bob.bio.face/bob/bio/face/test/test_baselines.py::test_attentionnet 

Well, I don't know what to do. Those are the faceXzoo models and we are using their code.

One thing we can do to optimize their code is to cap the head of the models off. They released them with the head (which occupies a big chunk of the total model size).

That would be a good thing to do. 16 GB of RAM for testing should be really the limit. I have 16 GB of RAM on both my laptop and virtual workstation and cannot get more RAM.