beat.core issueshttps://gitlab.idiap.ch/beat/beat.core/-/issues2019-10-04T08:44:08Zhttps://gitlab.idiap.ch/beat/beat.core/-/issues/89Improve output error messages2019-10-04T08:44:08ZSamuel GAISTImprove output error messagesWhen not enough data is written on an output, the related cache will be cleaned.
An error message must be shown to the user so he knows what is happening.When not enough data is written on an output, the related cache will be cleaned.
An error message must be shown to the user so he knows what is happening.https://gitlab.idiap.ch/beat/beat.core/-/issues/90Execute experiment in different environment2019-10-07T13:31:09ZSamuel GAISTExecute experiment in different environmentCurrently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of package...Currently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of packages.
While this can't be done with the local executor, it can be implemented using the subprocess executor.
The subprocess executor makes call to the scripts needed to run the algorithms/databases processes as would be with the docker executor. This fact can be used to run the scripts from another environment and thus will use the packages available there.
This will also simplify the creation of new execution environment and their transition do docker images.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/91LICENSE file location in conda recipe is wrong2019-11-14T16:07:45ZAndré AnjosLICENSE file location in conda recipe is wrongIt should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.It should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/92Simplifying toolchain connectivity2019-11-18T09:14:34ZAndré AnjosSimplifying toolchain connectivityThought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our user...Thought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our users a bit: why not avoiding this discrete port to port connection between blocks and accept "whole" block connections only?
The idea is to design toolchains only based on block connections instead of block input/outputs. As a result, a block that receives as an input data from another block, may access any of its outputs.
I'm not sure about the technical challenge(s) behind this, but that would definitely simplify the toolchain editor a bit.
My current understanding is that this does not affect block synchronization and may be relatively "easy" to implement.
@samuel.gaist: what do you think? Easy to roll out?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/93New environment descovery does not respect the raise on error property2020-01-29T13:15:19ZSamuel GAISTNew environment descovery does not respect the raise on error propertyIf the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call ...If the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call `beat editor refresh-env` on a machine without docker or with the daemon stopped.
A current workaround is to create the `.environments.json` file in the prefix with `{}` as its content.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/95Schema: missing constraints2020-03-10T16:08:31ZSamuel GAISTSchema: missing constraintsOn the platform, fields like the short descriptions have for example size constraints.
These should be reflected here to avoid issues when creating a new object.On the platform, fields like the short descriptions have for example size constraints.
These should be reflected here to avoid issues when creating a new object.https://gitlab.idiap.ch/beat/beat.core/-/issues/29Loops between blocks (MAP/Reduce)2020-05-13T06:32:34ZTiago de Freitas PereiraLoops between blocks (MAP/Reduce)For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And ...For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And you do this (Map/Reduce) several times.
To implement that using platform, you need two blocks, one for the Map and one for the Reduce.
Since the flow of the data in the platform is just forward, it is not possible to do a loop between these two blocks
Would be nice to have that feature.https://gitlab.idiap.ch/beat/beat.core/-/issues/97Created database v2 from template is wrong2020-06-18T13:58:35ZAmir MOHAMMADICreated database v2 from template is wrongI have created a v2 database (found in [prefix.tar.xz](/uploads/37186a98f55a1d68e7b316564b8ae85e/prefix.tar.xz))
but when I tried to finally run it from an experiment, I got:
```
$ beat exp run amohammadi/amohammadi/iris-pad-with-train/1...I have created a v2 database (found in [prefix.tar.xz](/uploads/37186a98f55a1d68e7b316564b8ae85e/prefix.tar.xz))
but when I tried to finally run it from an experiment, I got:
```
$ beat exp run amohammadi/amohammadi/iris-pad-with-train/1/iris-pad-with-train
Created cache path `/home/amir/idiap/git/beat/livedet2020-iris-beat-demo/prefix/cache'
Index for database livedet_iris_2020/1 not found, building it
Index for database livedet_iris_2020/1 not found, building it
Running `amohammadi/lbp-feature-extractor/1' for block `test_features'
Start the execution of 'amohammadi/lbp-feature-extractor/1'
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 123, in run
return getattr(obj, method)(*args, **kwargs)
TypeError: setup() got an unexpected keyword argument 'start_index'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/bin/beat", line 11, in <module>
sys.exit(main())
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/decorators.py", line 92, in _decorator
value = view_func(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 780, in run
return run_experiment(config, name, force, docker, local, environment, quiet)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/core/execution/local.py", line 360, in process
loop_socket=self.loop_socket,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/execution/algorithm.py", line 166, in __init__
databases=databases,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/helpers.py", line 248, in create_inputs_from_configuration
end_index=end_index,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/database.py", line 184, in setup
end_index=end_index,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 127, in run
six.reraise(exc, exc(value), traceback)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 123, in run
return getattr(obj, method)(*args, **kwargs)
RuntimeError: setup() got an unexpected keyword argument 'start_index'
^CException ignored in: <module 'threading' from '/home/amir/miniconda/envs/iris/lib/python3.7/threading.py'>
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/lib/python3.7/threading.py", line 1307, in _shutdown
lock.acquire()
KeyboardInterrupt
```Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/98local executer does not show the traceback of algorithms in case of failure.2020-06-29T11:54:07ZAmir MOHAMMADIlocal executer does not show the traceback of algorithms in case of failure.For example:
```
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packa...For example:
```
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/core/execution/local.py", line 405, in process
"Algorithm {} process failed".format(self.algorithm.name)
RuntimeError: Algorithm amohammadi/svm-classify-0-100/1 process failed
```
It's not very useful to know that the algorithm just failed. Having a traceback is more useful.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/99The outputs parameter from an Algorithm does not raise exceptions for unknown...2020-07-08T15:29:36ZAmir MOHAMMADIThe outputs parameter from an Algorithm does not raise exceptions for unknown outputsI have an algorithm, let's say this:
```python
class Algorithm:
def process(self, inputs, data_loaders, outputs):
# Writes the output
outputs["score"].write({"value": np.int32(60)})
return True
```
but the out...I have an algorithm, let's say this:
```python
class Algorithm:
def process(self, inputs, data_loaders, outputs):
# Writes the output
outputs["score"].write({"value": np.int32(60)})
return True
```
but the output is called `scores` and not `score`. However, the output of `outputs["score"]` is `None` instead of an exception, why?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/101Add tmpfs when running a container2020-07-29T09:18:53ZSamuel GAISTAdd tmpfs when running a containerDespite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file c...Despite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file creation coming from the entrypoint script loading the correct conda environment.
"Unsuccessfully failed" here means that the full experiment run was marked as good despite that error happening which in fact did not even allow for a proper start of the execution procedure.
The fix for this situation is to add a minimal writable tmpfs for /tmp in order to have the space necessary for that file to be created.
It might also be useful to have one for /run even if it does not look like it is being used currently (at least in the same way as /tmp seemed to not be used).Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/100Follow-up from "[livdet-iris-2020] Fix category type in view"2020-08-05T08:35:55ZAmir MOHAMMADIFollow-up from "[livdet-iris-2020] Fix category type in view"The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was li...The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was like this:
> ```
> $ beat exp run amohammadi/amohammadi/livdet/1/livedet
> Skipping execution of `amohammadi/livdet_baseline/1' for block `algorithm' - outputs exist
> Running `amohammadi/livdet_analyzer/1' for block `analyzer'
> Start the execution of 'amohammadi/livdet_analyzer/1'
> Block did not execute properly - outputs were reset
> Standard output:
>
> Standard error:
>
> Captured user error:
> File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/baseformat.py", line 115, in setup_scalar
> % (attrname, formatname, numpy.array(value).dtype, dtype)
> TypeError: cannot safely cast attribute `value' on dataformat `system/integer/1' with type `int64' to `int32' without precision loss
> Captured system error:
>
> Error: Error occured: returned value is 1
> Removing cache files: No data written
> ```
> which was very hard to debug.
> The problem here is that the backtrace error I was getting was not telling me this error was in the view and not in my analyzer (the step I was on when hitting this error). So, I kept looking at my analyzer and thinking why I get this error while the error was in the view.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/74Update the schemas to better reflect parameter type support in different objects2020-10-19T15:16:45ZJaden DIEFENBAUGHUpdate the schemas to better reflect parameter type support in different objectsSee https://gitlab.idiap.ch/beat/beat.editor/issues/205#note_40301
The issue we found first is that the algorithm editor's parameter logic doesn't actually support complex types, but the algorithm schema indicates support. See the algor...See https://gitlab.idiap.ch/beat/beat.editor/issues/205#note_40301
The issue we found first is that the algorithm editor's parameter logic doesn't actually support complex types, but the algorithm schema indicates support. See the algorithm schema, `algorithm/common.json:80`:
```json
80 "parameter": {
81 "type": "object",
82 "properties": {
83 "type": { "$ref": "../common/1.json#/definitions/basetype" },
84 "default": { "$ref": "../common/1.json#/definitions/value" },
85 "description": { "type": "string" }
86 },
87 "required": [
88 "type"
89 ]
90 },
```
The `type` field references the `basetype` types, which includes complex types, but the editors (`beat.web` & `beat.editor`) don't actually support complex types for algorithm parameters.
I'll be looking through the other schemas and will update this issue with other things I find, if any.
EDIT: didn't find anything else in the other objects' schemas parameter definitionsJaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.core/-/issues/59Tight "soft-loops" controlling algorithmic training2020-11-04T17:09:18ZAndré AnjosTight "soft-loops" controlling algorithmic trainingIn the context of the ALLIES project, a so-far-avoided requirement has popped up again the meeting: block loops.
I created this bug to differentiate from #29, because the requirements here are a bit different: in the use-case to be cons...In the context of the ALLIES project, a so-far-avoided requirement has popped up again the meeting: block loops.
I created this bug to differentiate from #29, because the requirements here are a bit different: in the use-case to be considered, platform users need to be able to train a model depending on the output of another block. The main research question of the project needs to answer is: is it possible to implement life-long learning systems (LLLS) that can be taught by a human and improve over time? The outcome of the project must be reproducible (therefore our involvement).
In order to simulate this, partners provided the following diagram:
![modes](/uploads/a4e1aa953cd2bced7fa834da5d300e63/modes.png)
On the left, you'll see conventional ways to "adapt" statistical models from new input data. Given the model and more data, the model can be adapted to new incoming information (a.k.a. "source" in these diagrams).
In the context of ALLIES, partners want to have reproducible toolchains that allow third-parties to create the training system on the right of this image. In this context, the yellow boxes under the "training" (cyan) boxes simulate a human-user interaction with the system. E.g.: it could be a human telling a machine translation system the translation of a sentence (or paragraph) needs to be corrected, alongside with the correction, or it could be a human telling a speaker diarization system that a certain segment belongs to a different speaker. Because humans have limited ability to provide corrections, the consortium wants to control the implementation and limits of the "yellow" boxes, while third-parties implement the "trainers" that go into the cyan boxes.
It is possible to "unroll" these loops for practical scenarios, but in order to test limits of LLLS, one can envisage hundreds if not thousand iterations which is not very practical to code in BEAT as of today.
One of the key questions while considering #29 was that of the cache: a system that loops will issue multiply different cache outputs which, using the current architecture, would have to be kept. In this new use-case, such a caching requirement is not present. Partners only want to ensure that the code in "yellow" boxes can be properly tagged by the consortium and that they can conduct searches and build leaderboards on experiments using those (and only those). This prevents implementors of the "cyan" training boxes to misinterpret the role and data from the simulated human (yellow boxes). As a consequence, **no caching is required within this loop**.Soft loopshttps://gitlab.idiap.ch/beat/beat.core/-/issues/104Experiment global properties regex is wrong for algorithms name2020-11-12T12:20:40ZSamuel GAISTExperiment global properties regex is wrong for algorithms nameThe `patternProperties` definition for algorithm related properties is missing `-` for the author part of the regex so it forbids creating experiments using valid algorithms.The `patternProperties` definition for algorithm related properties is missing `-` for the author part of the regex so it forbids creating experiments using valid algorithms.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/103Allow mounting of datasets folder on the algorithm container.2020-11-12T16:47:35ZSamuel GAISTAllow mounting of datasets folder on the algorithm container.In the context of the learn-real project, the training of robot related models requires the use of a very high number of images which can only be optimally loaded using PyTorch iterators.
This cannot be done currently as all data goes t...In the context of the learn-real project, the training of robot related models requires the use of a very high number of images which can only be optimally loaded using PyTorch iterators.
This cannot be done currently as all data goes through the database paradigm which cannot be used directly by PyTorch iterators.
Using the standard BEAT approach would lead to overly complex and inefficient implementation.
The solution proposed is that a new property can be inserted in the block configuration so that the root path of the database is also mounted on the algorithm container.
This will only work on algorithm containers that are connected to a database.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/105Refactor database sharing2020-11-17T12:09:23ZSamuel GAISTRefactor database sharingFollowing discussion on beat/beat.web!410, the database sharing feature will be refactored with the following:
- Add a new `direct_rawdata_access` optional field to the database schema
- Use that in place of the optional "share_databases...Following discussion on beat/beat.web!410, the database sharing feature will be refactored with the following:
- Add a new `direct_rawdata_access` optional field to the database schema
- Use that in place of the optional "share_databases" external field
- The field will apply to both v1 and v2 Database schema
- Support for optional field allows to keep the current databases workingSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/106ROS specific needs2020-11-30T08:50:02ZSamuel GAISTROS specific needsRequirement
===========
In the scope of the learn-real project, ROS is one of the possible system used however it needs to write some files (pids, small logs, etc.) in order to run properly however these can be discarded when the run en...Requirement
===========
In the scope of the learn-real project, ROS is one of the possible system used however it needs to write some files (pids, small logs, etc.) in order to run properly however these can be discarded when the run ends.
The current estimation done by @cazinn for the space needed is about 10Mb.
The ROS main folder is `~.ros` by default and that can be changed using the [ROS_HOME environment variable](http://wiki.ros.org/ROS/EnvironmentVariables#System_Data_Environment_Variables).
Current situation
=================
For security reasons, our containers are run in read-only mode with two small read/write tmpfs mounted on the containers: /tmp and /var. Each of which is allocated 500kb which is enough for the use of BEAT.
We do have support for adding temporary volumes in place as well as setting custom environment variables.
Possible solutions
==================
In order to run Docker containers that can execute ROS, we have several possibilities:
1) Mount a tmpfs volume on the default ROS home folder
2) Make either the `/tmp` or `/var` volumes bigger and point `ROS_HOME` in a subfolder in them
3) Add a new tmpfs volume in a separated known folder and point `ROS_HOME` to it
`ROS_HOME` can either be hardcoded during the image creation process or dynamically set when running the container.
Main questions
==============
For solution 1, from where should the information about the size of the target folder come from as well as the target location ?
For solution 2, `ROS_HOME` can be hardcoded as part of the image creation
For solution 3, same questions as for solution 1 with the possibility of hardcoding the environment variable as in solution 2.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/64Database schema and runtime improvements2021-01-29T09:46:59ZAndré AnjosDatabase schema and runtime improvementsDatabases were originally designed monolithically, assuming the environment where packages necessary to read their contents is fixed (and does not change), and the location of files to be used for the readout is fixed. As an after thoug...Databases were originally designed monolithically, assuming the environment where packages necessary to read their contents is fixed (and does not change), and the location of files to be used for the readout is fixed. As an after thought, templates were introduced without a proper formalisation and, as a consequence, templates with the same names may appear in various database JSON declarations, w/o necessarily being the same. This issue supersedes #25 and gathers all changes required for a revamp in this area:
- [x] Protocol and set templates should be separated from the database view declaration to avoid repetition and centralise "task-related" declarations. This will effectively simplify the declaration of new datasets
- [ ] The `root_db` parameter (maybe misspelled here) needs to be externalised as a runtime parameter during the run, as it is currently the case for other runtime prefixes such as algorithm caches. Currently, if one downloads the database view, they need to change this parameter by hand
- [x] The environment required to run the database view to provide the data for the experiment needs to be configured and have an entry on the database JSON declaration. This ensures that changes in the environment (docker image), will imply in new caches being generated at all times. As of today, it can happen that hashes are not regenerated even if the environment changes completely.
- [x] A default db env docker image (possibly named `beat.env.databases`) should be provided on docker hub. This avoids conflicts in future because when using multiple databases in one experiment, only one image can be used.
- [ ] The prototypes in `beat.core` should come configured to use this image.
- [ ] This new JSON entry must be documented in `beat/docs` and users should be recommended to use this image.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/107Error on docker client creation2021-04-13T12:14:32ZSamuel GAISTError on docker client creationAt least on macOS when the daemon is not running, the client creation may fail and thus lead to an exception.
This is a case that is not yet handled like the others that follows when looking up for images.
Related to beat/beat.editor#282At least on macOS when the daemon is not running, the client creation may fail and thus lead to an exception.
This is a case that is not yet handled like the others that follows when looking up for images.
Related to beat/beat.editor#282Samuel GAISTSamuel GAIST