Resolve "FAR and FRR thresholds are computed even when there is no data support"

Closes #27 (closed)

Also changes behavior of far_threshold and frr_threshold where the returned threshold guarantees the at most the requested far/frr value.

added 1 commit

• 374690d5 - Added a warning before returning NaN

Compare with previous version

unmarked as a Work In Progress

assigned to @amohammadi

@amohammadi Is this what you wanted?

added 1 commit

• 61237f50 - Add some obvious test cases

Compare with previous version

@mguenther I pushed some test cases here that were mentioned in #27 (closed) and I think these tests should pass. Let me know what do you think.

assigned to @mguenther

added 1 commit

• 5142b646 - Correct the threshold calculation implementation in far_threshold and…

Compare with previous version

@mguenther I have fixed the issue and also changed the behavior. You can still get the NaNs as discussed in #27 (closed) but I have changed the implementation that when a requested FAR or FRR is not possible, it will return a threshold that will give you a lower value in FAR and FRR than that. I asked around today and everybody seemed to believe that a lower or equal value FAR, FRR should be guaranteed. @andre.anjos @tiago.pereira @sbhatta were the people that I asked.

Basically I ended up moving through scores and calculating the FAR/FRR until it was bigger than requested. It might be slower but is much easier to understand and less error prone.

For example, when you are deploying a biometric system, you always need to make sure that your FAR and FRR value is lower than a certain amount not at least that value.

The code is not well written probably. If you have some code style comments, I would be happy if you push directly to this branch.

added 1 commit

• d9ff46cb - Fix comments [skip ci]

Compare with previous version

added 1 commit

• 89511eeb - rename check_index to index

Compare with previous version

@amohammadi Thanks for this suggestion. I wanted to avoid running through the scores, but I am now trying almost four hours to get the index correct in all corner cases...

changed the description

Yes, I have spent all my day on this. At some point I realized it was not going to work with your implementation and I had to implement from the beginning. Now, it may run slower but at least it is correct and the code is easy to understand.

added 1 commit

• 56f95d3e - Add more tests

Compare with previous version

added 2 commits

• 7e6a0704 - Changed test case for different corner cases
• fecf6b35 - Merge branch…

Compare with previous version

I think you forgot the corner case introduced by yourself, i.e., when the scores are actually separable -- in which case you might want to return the smallest positive for far_threshold and the largest negative for frr_threshold, or maybe a value slightly higher (smaller) than the highest (lowest) value. Just using FAR=0 and FRR=0 as corner cases is not enough.

I will push an adaptation of your test case that will fail here.