Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • bob.measure bob.measure
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 5
    • Issues 5
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • bobbob
  • bob.measurebob.measure
  • Issues
  • #37
Closed
Open
Issue created Feb 23, 2018 by Amir MOHAMMADI@amohammadiOwner

Plugin-based scripts for metrics and plots

we would like to add a script here:

$ bob measure

These scripts will load their subcommands from setuptools entrypoints; similar to how the bob ... loads its subcommands. This script will also help other packages to implement their evaluation scripts.

For example, we could have:

$ bob measure evaluate
# would print and plot according to FAR and FRR
$ bob bio evaluate
# would print and plot according to FMR and FNMR
$ bob pad evaluate
# would print and plot according to APCER and BPCER

We could also have commands that are specific to one plot or metric:

$ bob measure hter
$ bob measure hist
$ bob bio hist
$ bob pad hist
$ bob pad roc
$ bob face_icb2018 # would plot the figures of bob.paper.face_icb2018

It's desirable to be able to chain these commands:

$ bob measure hter hist roc det
$ bob bio hter hist roc

The original text of this issue is below

I would like to add two scripts here:

$ bob metrics
$ bob plots

These scripts will load their subcommands from setuptools entrypoints; similar to how the bob ... loads its subcommands.

Then, we can augment these scripts in bob.bio.base and bob.pad.base. For example, we could have:

$ bob metrics generic
# would print FAR and FRR
$ bob metrics bio
# would print FMR and FNMR
$ bob metrics pad
# would print APCER and BPCER

$ bob plots hist bio
# would plot histogram of genuines and zei
$ bob plots hist pad
# would plot histogram of bona-fide and PA
$ bob plots hist vuln
# would plot histogram of genuines, zei, and PA for vulnerability analysis

$ bob plots det bio
# would plot det curve with FMR and 1-FNMR

$ bob plots face_icb2018
# would plot the figures of bob.paper.face_icb2018

Any package could augment these with setuptools entrypoint and click while bob.measure could be a source of generic functions that other packages could take advantage of to easily implement these scripts.

What do you think?

I think this

  • Would encourage developers to create similar commands and re-use other's commands.
  • Would allow everybody to have its own plot script which easily discoverable by others.

I already have an implementation of this but it's in bob.pad.base. I think it would be better if it was in bob.measure.

I would like to possibly tackle other issues while doing this:

  • Having ISO conforming plots. bob.bio.base#60 (closed) bob.pad.base#1
  • Simplifying complicated scripts bob.bio.base#65 (closed)
  • Moving bob.bio metrics and plots out of bob.measure #25 (closed)
Edited Mar 12, 2018 by Amir MOHAMMADI
Assignee
Assign to
Time tracking