beat.core issueshttps://gitlab.idiap.ch/beat/beat.core/-/issues2020-08-05T08:35:55Zhttps://gitlab.idiap.ch/beat/beat.core/-/issues/100Follow-up from "[livdet-iris-2020] Fix category type in view"2020-08-05T08:35:55ZAmir MOHAMMADIFollow-up from "[livdet-iris-2020] Fix category type in view"The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was li...The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was like this:
> ```
> $ beat exp run amohammadi/amohammadi/livdet/1/livedet
> Skipping execution of `amohammadi/livdet_baseline/1' for block `algorithm' - outputs exist
> Running `amohammadi/livdet_analyzer/1' for block `analyzer'
> Start the execution of 'amohammadi/livdet_analyzer/1'
> Block did not execute properly - outputs were reset
> Standard output:
>
> Standard error:
>
> Captured user error:
> File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/baseformat.py", line 115, in setup_scalar
> % (attrname, formatname, numpy.array(value).dtype, dtype)
> TypeError: cannot safely cast attribute `value' on dataformat `system/integer/1' with type `int64' to `int32' without precision loss
> Captured system error:
>
> Error: Error occured: returned value is 1
> Removing cache files: No data written
> ```
> which was very hard to debug.
> The problem here is that the backtrace error I was getting was not telling me this error was in the view and not in my analyzer (the step I was on when hitting this error). So, I kept looking at my analyzer and thinking why I get this error while the error was in the view.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/92Simplifying toolchain connectivity2019-11-18T09:14:34ZAndré AnjosSimplifying toolchain connectivityThought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our user...Thought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our users a bit: why not avoiding this discrete port to port connection between blocks and accept "whole" block connections only?
The idea is to design toolchains only based on block connections instead of block input/outputs. As a result, a block that receives as an input data from another block, may access any of its outputs.
I'm not sure about the technical challenge(s) behind this, but that would definitely simplify the toolchain editor a bit.
My current understanding is that this does not affect block synchronization and may be relatively "easy" to implement.
@samuel.gaist: what do you think? Easy to roll out?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/76Clean up the schemas2019-06-02T02:18:24ZJaden DIEFENBAUGHClean up the schemas(@samuel.gaist this is the issue for our email discussion)
Both !55 and !65 showed that there were places to improve the schema. This issue expands on those MRs' work by using the v6/v7 json schema features (where appropriate) and addin...(@samuel.gaist this is the issue for our email discussion)
Both !55 and !65 showed that there were places to improve the schema. This issue expands on those MRs' work by using the v6/v7 json schema features (where appropriate) and adding tests to complicated parts of the schema. Right now I know of a couple of things that need to be done:
- [x] Use the `const` schema keyword instead of the `enum` with only one choice. This issue is documented in the schema, e.g. in beat.core/beat/core/schema/algorithm/3.json:
```json
115- "type": {
116: "$comment": "Change enum to const when tools allow v6 json schema",
117- "type": "string",
118- "enum": ["loop_user"]
119- }
--
144- "type": {
145: "$comment": "Change enum to const when tools allow v6 json schema",
146- "type": "string",
147- "enum": ["loop"]
148- }
```
- [ ] Use the `if`/`then`/`else` keywords from json schema v7 to represent more complex logic (e.g. the changes to algorithm parameters in !65)
- As found in !65, there are parts of the schema that aren't tested thoroughly. This is nebulous issue, however, so @flavio.tarsetti @samuel.gaist could you maybe specify what I should do here, so I don't go off the rails?
(As other issues are isolated, I'll add them to the list)Jaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.core/-/issues/74Update the schemas to better reflect parameter type support in different objects2020-10-19T15:16:45ZJaden DIEFENBAUGHUpdate the schemas to better reflect parameter type support in different objectsSee https://gitlab.idiap.ch/beat/beat.editor/issues/205#note_40301
The issue we found first is that the algorithm editor's parameter logic doesn't actually support complex types, but the algorithm schema indicates support. See the algor...See https://gitlab.idiap.ch/beat/beat.editor/issues/205#note_40301
The issue we found first is that the algorithm editor's parameter logic doesn't actually support complex types, but the algorithm schema indicates support. See the algorithm schema, `algorithm/common.json:80`:
```json
80 "parameter": {
81 "type": "object",
82 "properties": {
83 "type": { "$ref": "../common/1.json#/definitions/basetype" },
84 "default": { "$ref": "../common/1.json#/definitions/value" },
85 "description": { "type": "string" }
86 },
87 "required": [
88 "type"
89 ]
90 },
```
The `type` field references the `basetype` types, which includes complex types, but the editors (`beat.web` & `beat.editor`) don't actually support complex types for algorithm parameters.
I'll be looking through the other schemas and will update this issue with other things I find, if any.
EDIT: didn't find anything else in the other objects' schemas parameter definitionsJaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.core/-/issues/72Assert usage cleanup2019-05-09T13:26:49ZSamuel GAISTAssert usage cleanupFollowing bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.core.
Depending on how they are used, the following actions can be done:
1. Turn the statement into ...Following bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.core.
Depending on how they are used, the following actions can be done:
1. Turn the statement into an exception
2. Handle the case gracefully
3. Mark as not to get worried about
The current state of the code base is usually 3 which should be cleaned up.
All tests in this repository have been cleaned to use the testing framework corresponding toolshttps://gitlab.idiap.ch/beat/beat.core/-/issues/64Database schema and runtime improvements2021-01-29T09:46:59ZAndré AnjosDatabase schema and runtime improvementsDatabases were originally designed monolithically, assuming the environment where packages necessary to read their contents is fixed (and does not change), and the location of files to be used for the readout is fixed. As an after thoug...Databases were originally designed monolithically, assuming the environment where packages necessary to read their contents is fixed (and does not change), and the location of files to be used for the readout is fixed. As an after thought, templates were introduced without a proper formalisation and, as a consequence, templates with the same names may appear in various database JSON declarations, w/o necessarily being the same. This issue supersedes #25 and gathers all changes required for a revamp in this area:
- [x] Protocol and set templates should be separated from the database view declaration to avoid repetition and centralise "task-related" declarations. This will effectively simplify the declaration of new datasets
- [ ] The `root_db` parameter (maybe misspelled here) needs to be externalised as a runtime parameter during the run, as it is currently the case for other runtime prefixes such as algorithm caches. Currently, if one downloads the database view, they need to change this parameter by hand
- [x] The environment required to run the database view to provide the data for the experiment needs to be configured and have an entry on the database JSON declaration. This ensures that changes in the environment (docker image), will imply in new caches being generated at all times. As of today, it can happen that hashes are not regenerated even if the environment changes completely.
- [x] A default db env docker image (possibly named `beat.env.databases`) should be provided on docker hub. This avoids conflicts in future because when using multiple databases in one experiment, only one image can be used.
- [ ] The prototypes in `beat.core` should come configured to use this image.
- [ ] This new JSON entry must be documented in `beat/docs` and users should be recommended to use this image.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/57Is `return True` needed in the algorithms API still?2018-05-30T14:33:58ZJaden DIEFENBAUGHIs `return True` needed in the algorithms API still?Not sure where I should be putting this issue...
Right now, as far as I know, `process` methods in algorithms must return `True`. Is this still required?
@samuel.gaistNot sure where I should be putting this issue...
Right now, as far as I know, `process` methods in algorithms must return `True`. Is this still required?
@samuel.gaistSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/56Docker timer tests rely on having adequate CI resources2018-12-12T09:13:59ZJaden DIEFENBAUGHDocker timer tests rely on having adequate CI resourcesSometimes docker timing tests will fail when the tests go too slowly. These tests rely on a hard-coded number of seconds/milliseconds with numbers that are reasonable to assume on one's own machine but obviously aren't designed for a CI ...Sometimes docker timing tests will fail when the tests go too slowly. These tests rely on a hard-coded number of seconds/milliseconds with numbers that are reasonable to assume on one's own machine but obviously aren't designed for a CI (just see the [docker tests](https://gitlab.idiap.ch/beat/beat.core/blob/85cef289db0eac7aaa47308dd92cc5003a6a689b/beat/core/test/test_docker.py).
Some of these tests can just be changed. For example, the `test_images_cache` tests are actually relative tests - you just want to be sure that doing something a second time is faster than the first time. I believe that the CI performance would be consistent across each of these tests (which shouldn't last for more than a couple seconds).
There's other ones though, like the timeout tests. Not sure what to do with those.
This doesn't unconditionally make the CI fail since rerunning the job usually fixes it.https://gitlab.idiap.ch/beat/beat.core/-/issues/49Update the plotterparameter schema to associate it to a plotter2018-04-17T07:55:49ZJaden DIEFENBAUGHUpdate the plotterparameter schema to associate it to a plotterRight now there's no way to tell via the naming scheme or the plotterparameter JSON which plotter the parameter is associated to. The easiest way would be to add a field, say "plotter", which has the associated plotter object's name. See...Right now there's no way to tell via the naming scheme or the plotterparameter JSON which plotter the parameter is associated to. The easiest way would be to add a field, say "plotter", which has the associated plotter object's name. See beat.cmdline#20 for the discussion that led up to this.
The idea could be something like what I suggested in the other thread:
```json
{
"plotter": "plot/bar/1",
"grid": true,
"title": "DET ISO/IEC 19795-1:2006",
"title-fontsize": 12,
"det": true,
"legend-loc": "best",
"width": 800,
"height": 600,
"dpi": 120,
"legend-fontsize": 8,
"xlim-left": 0,
"xlim-right": 40,
"ylim-bottom": 0,
"ylim-top": 40,
"axis-fontsize": 10
}
```
The "plotter" field stores the reference to the associated plotter.
However, it might be best to use a field name that the user couldn't accidentally use, such as `#plotter`.
@samuel.gaist @flavio.tarsetti @andre.anjos what do you all think? This is sorta blocking time-sensitive work so it'd be better to decide quickly.https://gitlab.idiap.ch/beat/beat.core/-/issues/48Enum types2018-03-15T08:41:59ZJaden DIEFENBAUGHEnum typesIt would be nice to have enum types supported, like in the algorithm parameters.It would be nice to have enum types supported, like in the algorithm parameters.https://gitlab.idiap.ch/beat/beat.core/-/issues/43Statistics output by agent don't account for the CPU time spent reading data ...2017-10-16T09:00:18ZAndré AnjosStatistics output by agent don't account for the CPU time spent reading data from diskThis is inconsistent with our cpulimit policy, for which we are stipulating both the agent and the user process are bound to the same CPU limit and must share the resources.
In this way, it would be best if we'd also account the CPU u...This is inconsistent with our cpulimit policy, for which we are stipulating both the agent and the user process are bound to the same CPU limit and must share the resources.
In this way, it would be best if we'd also account the CPU usage from the agent + the user process on the final statistics output.
One must be careful as the agent is called in two different contexts (`beat.scheduler iodaemon` or `beat.cmdline exp run`), but the accounting must always come out right.https://gitlab.idiap.ch/beat/beat.core/-/issues/36Analyzer should not be able to write more than one output2018-12-19T13:58:55ZAndré AnjosAnalyzer should not be able to write more than one outputCurrently, analyzers running on the platform can write as many outputs as times the `process()` function is called on the user code. This is non-sensical and ends-up creating invalid results for an experiment.
The platform should trig...Currently, analyzers running on the platform can write as many outputs as times the `process()` function is called on the user code. This is non-sensical and ends-up creating invalid results for an experiment.
The platform should trigger an error by the second time the user tries to write a result in an analyzer.Philip ABBETPhilip ABBEThttps://gitlab.idiap.ch/beat/beat.core/-/issues/33Thorough testing and API support for plotters and plotter parameters2018-12-19T14:24:41ZAndré AnjosThorough testing and API support for plotters and plotter parametersAs of today, the concept of a `plotter` is implemented in beat.core. It would be good to extend its implementation in the core so that:
1. Plotters can be downloaded/uploaded/edited using the web API and via the command-line tool
2. ...As of today, the concept of a `plotter` is implemented in beat.core. It would be good to extend its implementation in the core so that:
1. Plotters can be downloaded/uploaded/edited using the web API and via the command-line tool
2. A command-line utility is made available to plot a given experiment output
3. Plotter parameters are instantiated in the core, so that we can also apply them while locally plotting
This should be done after the FG tutorial.https://gitlab.idiap.ch/beat/beat.core/-/issues/29Loops between blocks (MAP/Reduce)2020-05-13T06:32:34ZTiago de Freitas PereiraLoops between blocks (MAP/Reduce)For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And ...For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And you do this (Map/Reduce) several times.
To implement that using platform, you need two blocks, one for the Map and one for the Reduce.
Since the flow of the data in the platform is just forward, it is not possible to do a loop between these two blocks
Would be nice to have that feature.