Plugin-based scripts for metrics and plots

Yes, please!

Hey,

I have one possible candidate to contribute to this one. I have an entry point in the htface package that averages the CMC, ROC and RR for crossvalidation evaluation (one protocol with several folds)

@amohammadi I like your idea. This would make it much easier to distribute plotting functionality.

I have just one remark: I guess it would be easier and better understandable if you change the order of arguments slightly. Instead of

$ bob plots hist bio

you should have

$ bob plots bio hist

I assume that this will be also easier to implement, i.e., having all bob plots bio to be implemented in bob.bio.base, and alike. Then each function can define, which kinds of plots it supports, e.g., there is no use of a CMC curve for PAD algorithms.

I have a small concern when it comes to large score files, which would need to be read and interpreted twice if two different plots (e.g., ROC and DET) should be plotted. If you know a quick solution to have several plots in one call to bob bio plot, you might want to implement that. Otherwise, you can ignore this concern.

Having scores loaded once means the command line parameters need to be fixed and done by one command which could lead to tons of options in one command.

Having scores loaded several times and several different commands invoked separately makes the job tiresome and slow but keeps the number of options minimum and the commands are easier to understand.

I think we can have both here. At least in bob.measure there should not be any restriction on that.

mentioned in issue #38 (closed)

Genuines = Bonafide

@andre.anjos Bonafide = Bona Fide ^^

@amohammadi My concern was not to be taken too seriously. I think, we can live with just a single plot per function call. If you really want to plot different things and our implementation is too slow, you can always implement things yourself.

Couple of points:

Plotting is easy but customization of plots is not! I am trying to figure out a trade-off between the number of options that we should have for plot scripts. Of course, with a plugin-based system, everyone can have its own plotting command but we would want to provide good enough defaults with enough and reasonable customizations at the same time too. Any ideas on what should be customizable and what not? So far I have identified these as common options (for plots):

# read this code snippet from bottom to top

def common_options(f):
    # more import options go down the list here.
    f = click.pass_context(f)
    f = verbosity_option()(f)
    f = click.option(
        '--style', multiple=True, type=click.types.Choice(plt.style.available),
        help='The matplotlib style to use for plotting. You can provide '
        'multiple styles by repeating this option')(f)
    f = click.option(
        '--titles', help='The title for each system comma separated. '
        'Example: --titles ISV,CNN')(f)
    f = click.option(
        '--top', type=FLOAT,
        help='To give to ``plt.subplots_adjust(top=top)``. If given, first '
        'plt.tight_layout is called. If you want to tight_layout to be called,'
        ' then you need to provide this option.')(f)
    f = click.option(
        '--legend-ncol', default=3, show_default=True,
        type=INT,
        help='The number of columns of the legend layout.')(f)
    f = click.option(
        '--figsize', help='If given, will run '
        '``plt.figure(figsize=figsize)(f)``. Example: --fig-size 4,6')(f)
    # f = click.option(
    #     '--y2-label',
    #     help='The id of figures which should have y2_label separated by '
    #     'comma. For example ``--y2-label 1,2,4``.')(f)
    f = click.option(
        '--y1-label',
        help='The id of figures which should have y1_label separated by '
        'comma. For example ``--y1-label 1,2,4``.')(f)
    f = click.option(
        '--x-label',
        help='The id of figures which should have x_label separated by '
        'comma. For example ``--x-label 1,2,4``.')(f)
    f = click.option(
        '--subplot', type=INT, default=111,
        show_default=True, help='The order of subplots.')(f)
    f = click.option(
        '-o', '--output', type=File(mode='w'),
        default='plots.pdf', show_default=True,
        help='The file to save the plots in.')(f)
    f = click.argument('scores', nargs=-1)(f)
    return f

I have found myself using both bob_compute_perf.py (in bob.measure) and evaluate.py (in bob.bio.base). Usually, when I am working on one system, I want to get all the information possible and so far bob_compute_perf.py has been the best option. It will give you all the possible plots and numbers on one system so you can quickly gauge what is going on. However, when I want to compare several systems, I use evaluate.py (because bob_compute_perf.py can't do that). I think when comparing several systems, you only want to see one metric? (like ROC) or you still want to compare all systems with all the metrics? What do you think?
At some point @mguenther was suggesting to accept configuration files for options in bob.bio.base#65 (comment 20673) Although I like the idea now, comes the question on what should be the command-line's arguments? For example, which of the below?:

$ evaluate.py config1.py config2.py --dev-files ... --eval-files ...
$ evaluate.py scores...{dev,eval} --config config1 config2

Because for quick and dirty (without config files) plotting providing --dev-files and --eval-files is cumbersome but when you want to pass the list of scores in the config file too your command will look like bob roc --config config.py which makes you think that this --config option is redundant. What do you think? So far I am leaning towards not accepting config files at all because it makes both writing easy and the usage easily understandable.

I am thinking instead of having bob metrics and bob plots, we can just have bob measure. It might make it easier to know where this root command is coming from and there is really no advantage in having two commands here.

mentioned in issue #39 (moved)

You might want to add the following options:

--legend-location: The location of the legend, might be None by default
--color-order: An order of colors (might be tricky to implement, though)
--line-width: the line width of the lines to plot
--marker-styles: the markers (if any) to use in your plots

I am not sure, what you mean with --style, you should be more explicit. I also don't know how the --subplot option is used, or what that means.

I agree with the point of having only one bob measure command.

As a part of the bob 4 release and to answer issues #25 (closed), #37 (closed), #38 (closed) and improve code architecture and maintenance, I suggested the following modifications:

Plotting core: at the moment, plots are implemented a little bit everywhere with sometimes some redundancy. We also would like to have generic plotting functionalities in bob.measure and specialized plots in specific modules (e.g. bob.bio.base) without duplicating code. For that purpose, I propose to abstract the plotting (and data/results access) functionalities as illustrated in the following class diagram: The idea is to have the abstract class GraphicrepResentation and the generic derived classes (e.g. Plot2D) in bob.measure and other package specific plots in the corresponding package. Note that the generic derived class in bob.measure should be functional. Thus, while implementing specific plots derived it should be possible to use most of the functionalities of the generic base plot and avoid code duplication (note that the propose mechanism may vary but the goal should remain the same). In theory, most of the plotting library calls (e.g. matplotlib) should remain in the base classes so that the maintenance and evolution of the code is easier (also if you want to use another library one day).
Data provider: to plot things we need data. In order to avoid having codes that read specific package related files in bob.measure, I also suggest to abstract read/write access of the files and data in general as presented in the following Figure: As before, the idea is to have the abstract and generic providers in bob.measure and implement derived specifics providers in the different modules. The role of the provider is to feed raw data (e.g. dev-scores) to the GraphicRepresentation. As illustrated in the Figure, source of data is no necessary files which could be useful for future bob/beat compatibility/integration.
The quantities (e.g. FAR, FRR) that are actually displayed in the graphics are then computed from the raw data provided by the provider. In the first Figure, this is performed by the preprocessData() method. Alternatively, as for data providers, an abstract object Process handling the processing could be stored in the base class and instead of implementing the preprocessData(), we implement derived class of Process (which follows the Strategy design pattern). In both case, the existing processing function (e.g. epc C++ implementation) can be used.
Points 1 to 4 describe the core of the plotting. Then APIs to actually plot the graph objects should be implemented. Each module defining its own GraphicalRepresentation should define (in for example plot_api.py) a corresponding API function to generate the plot. These function should use Click as proposed by @amohammadi in his previous comment. We can then discuss the amount of flexibility we want and where it should be. For example there would be two strategies: give a lot of flexibility through the command options or by deriving extremely specialized classes. Both have pros and cons. My preference however goes to the first option: I think we should limit the amount of derived GraphicalRepresentations so that we actually know what they are doing. Moreover, when too many command line options are present, we can use configuration files and of course, most of the options would have default values.
With the previous settings, only one plot at a time. In my opinion (but with no experience with bob), it may be suitable to have predefined Experiments where (similarly to evaluate.py) several plots and measures can be computed simultaneously (we could also think of some optimization so that the same quantities are not computed multiple times). It would be (I think) easy to compile the click options of the different plots for those Experiments.

In summary, by following this architecture we have:

a better separation between modules
a more modular way of defining new plots without duplicating code
a flexible way to use plotting functionalities (single plot or full Experiment)

As I my experience with Bob is very limited, I probably missed things, so please @amohammadi @mguenther @sebastien.marcel @andre.anjos @tiago.pereira @samuel.gaist and others, let me know what you think.

@theophile.gentilhomme thank you for the analysis and proposal -- it is great to have you on this effort and I am sure you will get some comments !

Hey @theophile.gentilhomme, thanks for the work.

Although I was not part of the discussions about this, follow some of my thoughts:

1 - IMHO, I think we don't need to abstract the whole plotting. Matplotlib is pretty standard and I don't see us changing the plot backend (unless this will connect with BEAT at some point). I don't know what others think about this one.

2 - The Data provider seems a good thing to have. As of today we have our 4/5 columns text format for our biometric recognition stuff (which has its respective reader in bob.measure). The group is growing and there are people working with new tasks (PAD, Diarization, etc..), which may require some specific format to read/write the scores.

3 - About the entrypoints for the plots; as long as we have something in the way @amohammadi described in the beginning of this post (very granular and with an incremental approach), I'm happy with it. In this way, will be easier for us to customize and share our plot scripts.

Cheers

Hey @tiago.pereira

I strongly disagree with your arguments. The fact that matploblib is currently used by bob is, or rather should be, an implementation detail. Doing as you propose excludes the possibility of testing any other library or plotting solution without copying the whole module and re-implement everything. You deny bob's users an easy entry to use their own solution if they have any and for them to provide new backends if they'd like to. You also enter the territory of maintenance nightmare because the day you want to add a new plotting solution, you'll have to break everything to integrate it meaning that you would also break all your users setup.

In the absolute, matplotlib should even be an optional dependency enabled by the user. It would make bob's footprint way lighter.

Hi, thank you for coming up with this proposal. Here are my comments:

Different plots require different types and numbers of scores. For example, to plot an ROC curve, you need a set of negative and positive score arrays. https://www.idiap.ch/software/bob/docs/bob/docs/master/bob.measure/doc/py_api.html#curves . To plot an EPC curve, you need 4 score arrays: dev and eval times negatives and positives. To plot an EPSC, you need 6 score arrays: {dev,eval}*{negatives1,negatives2,positives}. I don't see how this ProviderBase is going to work for all plots. (You see I am talking about score arrays here, see point 2)
Although the data provider is a good thing to have, our functions already take array_like representations of scores and use them as if they were numpy arrays. I don't see how a MemoryProvider is going to help us. Maybe I have not understood your design.

a corresponding API function to generate the plot. These function should use Click as proposed

I don't want bob.measure.plot functions be integrated with Click. They should be pure functions like the ones we already have. Click should only be used when you want to implement the command line scripts.

In the absolute, matplotlib should even be an optional dependency enabled by the user. It would make bob's footprint way lighter.

That is already the case. Most of Bob work without matplotlib (unless you really want to plot something). If you see otherwise, please open a separate issue.

You deny bob's users an easy entry to use their own solution if they have any and for them to provide new backends if they'd like to.

That's not true. Bob.measure is written in C++ (See bob.measure.roc for an example), you cannot even use matplotlib there. Bob.measure produces the x-axis and y-axis (sometimes z-axis) numbers that you need to plot something in mainly C++ without depending on anything else.

Then, comes the actual plotting, this is the bob.measure.plot API which is using the bob.measure API and matplotlib to do the actual plotting. So in a way matplotlib is an optional dependency which leads to point 3:

3 . Matplotlib already comes with an object-oriented API https://matplotlib.org/api/pyplot_summary.html#the-object-oriented-api and https://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes which looks like what you are doing with your GraphicRepresentation class. Matplotlib by itself is an abstraction of plotting for Python and it comes with different backend implementations (AGG, GTK, QT): https://matplotlib.org/api/index_backend_api.html and many other Python libraries build upon matplotlib. So unless your GraphicRepresentation class inherits from matplotlib.axes.Axes or you actually have a matplotlib alternative in mind that is valid, I don't see a lot of gain in abstracting the plotting. I see the good design choice in it though but just saying we might use this in BEAT (where BEAT uses matplotlib too: https://gitlab.idiap.ch/beat/beat.core/blob/master/setup.py) does not justify it for me. Keep in mind that bob.measure.plot API is very simple and all functions there are just two lines of code: https://gitlab.idiap.ch/bob/bob.measure/blob/c89e5cb366cd903027b404e3b0d54d98065f8413/bob/measure/plot.py#L142

3 - About the entrypoints for the plots; as long as we have something in the way @amohammadi described in the beginning of this post (very granular and with an incremental approach), I'm happy with it. In this way, will be easier for us to customize and share our plot scripts.

@tiago.pereira I have done a revision of the commandline interface design which is available in this comment: #38 (comment 26050) which is not very granular anymore. I think it's a good trade-off between our concerns. If you are not happy with that, let's discuss this.

Hi Amir,

I think you missed some of the points: as described in the proposal, (pre)processing of the inputs would be done by proprecessData(). Thus, for plots that are very similar but require different processing of the data, only (mainly) this function needs to be implemented.
MemoryProvider is a dummy/lazy class that is used to pass an array. Also, it can be interesting to have this kind of provider when we actually do not want to write information between different tasks. For example, a (probably) more sophisticated memory provider could be used for integrating the beat core in bob, as in beat writing is strictly forbidden (alternatively we could think of a encrypted file base provider, but that is another discussion). But for the time being, and in accordance to the proposed architecture, it would be just the proper way of passing an array. Of course, it is still possible to use convenient functions in the base class that take arrays and internally generate the provider. But we still need this dummy provider to ensure the abstraction.

I don't want bob.measure.plot functions be integrated with Click.

I am not sure to understand why. IMHO, it would be useful to have functional plots in `bob.measure`, not only some abstract things you just use in other packages. And I do not see why click documented functions could not be used in macro scripts `Experiment`. But it is true that click commands cannot be used as usual functions anymore unless you use some tricks, so I guess would would need specific click command in bob.measure and leave functions as in `bob.measure.plot`.

That is already the case. Most of Bob work without matplotlib (unless you really want to plot something). If you see otherwise, please open a separate issue.

I will also leave @samuel.gaist answer this, but the problem is already in your statement " unless you really want to plot something" : if you want specific plots for each package to be implemented into this package, you will get matplotlib dependency everywhere. Of course, if there is no specific plots for the different modules and everything can use generic plots of bob.measure, the abstraction of the plotting might not be as useful. However, I think provider is a good solution in any-case.

I understand that most of the plots are implemented in two lines, very similar for each function. Of course with the proposed architecture, we could avoid to repeat these two lines or more for each graph, but I agree it would not justify by itself an new design. However, in addition to a better design and easier maintenance, I think it can help avoid code duplication and help unify the style and design of (new) graphs. For example, is not that true (I new to the field...) that ROC or DET are generic representations but can have specific displays in one field ? Also, your current plots uses basic matplotlib features. What if you want to have more sophisticated, fine tuned default representations? Also, it would be nice IMHO to have function graphs in bob.measure and specialized them in other packages without duplicating the code. And finally, yes, if you want to change your plotting library, abstraction would be extremely helpful.
which is not very granular anymore. I think it's a good trade-off between our concerns.

But isn't it better if we can have both?

That is already the case. Most of Bob work without matplotlib (unless you really want to plot something). If you see otherwise, please open a separate issue.

Then that's great and the situation may have improved since the last time I checked. From a cursory look there are still two packages that might prove problematic but it's unrelated to this repo and bug report. Since I'd say this is a more "overall" general issue, what would be the best place for such a report ?

Then that's great and the situation may have improved since the last time I checked. From a cursory look there are still two packages that might prove problematic but it's unrelated to this repo and bug report. Since I'd say this is a more "overall" general issue, what would be the best place for such a report ?

You can open open specific issues to your problem on each package and open a meta issue in bob/bob to track them.

My input to this saga...

IMO, it looks like a lot of work to replace matplotlib by abstracting it. I worry we'll very likely end up forwarding all minor bits from our API to matplotlib's to get stuff working - there will likely be a lot of "abstract" API duplication and, in the end, a dynamic dependence to its functionality. While I think that the core of the idea is nobel, IMO it will add a prohibitive maintenance cost that will befall on us, once @theophile.gentilhomme is allocated to other project. I'll compare this as doing the same for numpy. We, unfortunately, due to cost and time - will have to do hard choices here. This will probably mean, unless anyone has a better idea, to focus on matplotlib and define that as a fundamental building block of Bob. This will also simplify the life of @theophile.gentilhomme and allow him to help us on other parts of Bob.

I like the idea of the file provider, though I'd like to know a bit more of how things are going to work in practice (programmatic pieces of code with examples), given such files provide very often, very different information. All trials to create homogeneous interfaces did fail on the past.

I am not sure to understand why. IMHO, it would be useful to have functional plots in bob.measure, not only some abstract things you just use in other packages. And I do not see why click documented functions could not be used in macro scripts Experiment. But it is true that click commands cannot be used as usual functions anymore unless you use some tricks, so I guess would would need specific click command in bob.measure and leave functions as in bob.measure.plot.

You answered yourself :) click is more likely to die compared to matplotlib and it makes those functions unusable elsewhere.

But isn't it better if we can have both?

Of course we can have both. That's the whole idea of this plugin-based scripts issue. To start though, I think we can live with #38 (comment 26050) I think it's a good idea to have both too because it would force your implementation to be more modular but that would depend on how much time you have left.

Thank you all for putting your comments into this. @andre.anjos added his input too which is really good.

I have talked to people here and we think this issue is something of a small issue. It's good for @theophile.gentilhomme to start with this to get started on Bob but we want this task to be completed soon. We have more pressing issues in Bob that we think Theophile's time is more appreciated there.

I have updated my comment on #38 (comment 26050) to add an item for the generic plotting script (This issue #37 (closed)).

To summarize:

Please:

don't do GraphicRepresentation.
Keep the plotting functions bob.measure.plot as is (no classes, no click integration)
Do not add something like MemoryProvider. Python already has this concept of array_like inputs which already work in Bob.
We just need a base class that reads scores from a file or somewhere else and returns array_like objects. So maybe you can rename Provider to Reader. This will only be needed in implementing the bob measure script. bob.measure curves API will still work with array_like objects.
The tasks in #38 (comment 26050) are the main goal here.

Please open merge requests as soon as you start implementing something and we can iterate on it.

Questions for @theophile.gentilhomme :

How long do you think this is going to take? (Preferably less than a month)
Is it possible to use click's multi command chaining? http://click.pocoo.org/6/commands/#multi-command-chaining so that we can have something like bob measure roc .../scores and bob measure hter roc det epc .../scores. I don't think this is easily possible but if it was possible there would no difference between having one command that does everything and several commands that do specific things.

Thank you.

Plugin-based scripts for metrics and plots

Designs

Child items ...

Activity