beat.core issueshttps://gitlab.idiap.ch/beat/beat.core/-/issues2022-02-24T09:03:39Zhttps://gitlab.idiap.ch/beat/beat.core/-/issues/108Nightlies are failing for several reasons2022-02-24T09:03:39ZSamuel GAISTNightlies are failing for several reasonsThe nightlies are currently failing for several unrelated reasons:
- Looks like some keys have gone missing
- Some tests cannot be run because of database environment missing.
The first one is likely related to the use of cgroup v2 bas...The nightlies are currently failing for several unrelated reasons:
- Looks like some keys have gone missing
- Some tests cannot be run because of database environment missing.
The first one is likely related to the use of cgroup v2 based on the name of the fields missing and the information given on the [Container Stats information for the Docker v1.41 API](https://docs.docker.com/engine/api/v1.41/#operation/ContainerStats).
As for the database environments, the likely reasons is that all the runners and test machines had previous versions of the database images and thus the tests were run successfully although the test image pull method used later versions of them.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/107Error on docker client creation2021-04-13T12:14:32ZSamuel GAISTError on docker client creationAt least on macOS when the daemon is not running, the client creation may fail and thus lead to an exception.
This is a case that is not yet handled like the others that follows when looking up for images.
Related to beat/beat.editor#282At least on macOS when the daemon is not running, the client creation may fail and thus lead to an exception.
This is a case that is not yet handled like the others that follows when looking up for images.
Related to beat/beat.editor#282Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/106ROS specific needs2020-11-30T08:50:02ZSamuel GAISTROS specific needsRequirement
===========
In the scope of the learn-real project, ROS is one of the possible system used however it needs to write some files (pids, small logs, etc.) in order to run properly however these can be discarded when the run en...Requirement
===========
In the scope of the learn-real project, ROS is one of the possible system used however it needs to write some files (pids, small logs, etc.) in order to run properly however these can be discarded when the run ends.
The current estimation done by @cazinn for the space needed is about 10Mb.
The ROS main folder is `~.ros` by default and that can be changed using the [ROS_HOME environment variable](http://wiki.ros.org/ROS/EnvironmentVariables#System_Data_Environment_Variables).
Current situation
=================
For security reasons, our containers are run in read-only mode with two small read/write tmpfs mounted on the containers: /tmp and /var. Each of which is allocated 500kb which is enough for the use of BEAT.
We do have support for adding temporary volumes in place as well as setting custom environment variables.
Possible solutions
==================
In order to run Docker containers that can execute ROS, we have several possibilities:
1) Mount a tmpfs volume on the default ROS home folder
2) Make either the `/tmp` or `/var` volumes bigger and point `ROS_HOME` in a subfolder in them
3) Add a new tmpfs volume in a separated known folder and point `ROS_HOME` to it
`ROS_HOME` can either be hardcoded during the image creation process or dynamically set when running the container.
Main questions
==============
For solution 1, from where should the information about the size of the target folder come from as well as the target location ?
For solution 2, `ROS_HOME` can be hardcoded as part of the image creation
For solution 3, same questions as for solution 1 with the possibility of hardcoding the environment variable as in solution 2.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/105Refactor database sharing2020-11-17T12:09:23ZSamuel GAISTRefactor database sharingFollowing discussion on beat/beat.web!410, the database sharing feature will be refactored with the following:
- Add a new `direct_rawdata_access` optional field to the database schema
- Use that in place of the optional "share_databases...Following discussion on beat/beat.web!410, the database sharing feature will be refactored with the following:
- Add a new `direct_rawdata_access` optional field to the database schema
- Use that in place of the optional "share_databases" external field
- The field will apply to both v1 and v2 Database schema
- Support for optional field allows to keep the current databases workingSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/104Experiment global properties regex is wrong for algorithms name2020-11-12T12:20:40ZSamuel GAISTExperiment global properties regex is wrong for algorithms nameThe `patternProperties` definition for algorithm related properties is missing `-` for the author part of the regex so it forbids creating experiments using valid algorithms.The `patternProperties` definition for algorithm related properties is missing `-` for the author part of the regex so it forbids creating experiments using valid algorithms.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/103Allow mounting of datasets folder on the algorithm container.2020-11-12T16:47:35ZSamuel GAISTAllow mounting of datasets folder on the algorithm container.In the context of the learn-real project, the training of robot related models requires the use of a very high number of images which can only be optimally loaded using PyTorch iterators.
This cannot be done currently as all data goes t...In the context of the learn-real project, the training of robot related models requires the use of a very high number of images which can only be optimally loaded using PyTorch iterators.
This cannot be done currently as all data goes through the database paradigm which cannot be used directly by PyTorch iterators.
Using the standard BEAT approach would lead to overly complex and inefficient implementation.
The solution proposed is that a new property can be inserted in the block configuration so that the root path of the database is also mounted on the algorithm container.
This will only work on algorithm containers that are connected to a database.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/101Add tmpfs when running a container2020-07-29T09:18:53ZSamuel GAISTAdd tmpfs when running a containerDespite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file c...Despite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file creation coming from the entrypoint script loading the correct conda environment.
"Unsuccessfully failed" here means that the full experiment run was marked as good despite that error happening which in fact did not even allow for a proper start of the execution procedure.
The fix for this situation is to add a minimal writable tmpfs for /tmp in order to have the space necessary for that file to be created.
It might also be useful to have one for /run even if it does not look like it is being used currently (at least in the same way as /tmp seemed to not be used).Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/100Follow-up from "[livdet-iris-2020] Fix category type in view"2020-08-05T08:35:55ZAmir MOHAMMADIFollow-up from "[livdet-iris-2020] Fix category type in view"The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was li...The following discussion from beat.examples!25 should be addressed:
- [ ] @amohammadi started a [discussion](https://gitlab.idiap.ch/beat/beat.examples/-/merge_requests/25#note_53755): (+3 comments)
> btw the error for this was like this:
> ```
> $ beat exp run amohammadi/amohammadi/livdet/1/livedet
> Skipping execution of `amohammadi/livdet_baseline/1' for block `algorithm' - outputs exist
> Running `amohammadi/livdet_analyzer/1' for block `analyzer'
> Start the execution of 'amohammadi/livdet_analyzer/1'
> Block did not execute properly - outputs were reset
> Standard output:
>
> Standard error:
>
> Captured user error:
> File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/baseformat.py", line 115, in setup_scalar
> % (attrname, formatname, numpy.array(value).dtype, dtype)
> TypeError: cannot safely cast attribute `value' on dataformat `system/integer/1' with type `int64' to `int32' without precision loss
> Captured system error:
>
> Error: Error occured: returned value is 1
> Removing cache files: No data written
> ```
> which was very hard to debug.
> The problem here is that the backtrace error I was getting was not telling me this error was in the view and not in my analyzer (the step I was on when hitting this error). So, I kept looking at my analyzer and thinking why I get this error while the error was in the view.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/99The outputs parameter from an Algorithm does not raise exceptions for unknown...2020-07-08T15:29:36ZAmir MOHAMMADIThe outputs parameter from an Algorithm does not raise exceptions for unknown outputsI have an algorithm, let's say this:
```python
class Algorithm:
def process(self, inputs, data_loaders, outputs):
# Writes the output
outputs["score"].write({"value": np.int32(60)})
return True
```
but the out...I have an algorithm, let's say this:
```python
class Algorithm:
def process(self, inputs, data_loaders, outputs):
# Writes the output
outputs["score"].write({"value": np.int32(60)})
return True
```
but the output is called `scores` and not `score`. However, the output of `outputs["score"]` is `None` instead of an exception, why?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/98local executer does not show the traceback of algorithms in case of failure.2020-06-29T11:54:07ZAmir MOHAMMADIlocal executer does not show the traceback of algorithms in case of failure.For example:
```
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packa...For example:
```
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/core/execution/local.py", line 405, in process
"Algorithm {} process failed".format(self.algorithm.name)
RuntimeError: Algorithm amohammadi/svm-classify-0-100/1 process failed
```
It's not very useful to know that the algorithm just failed. Having a traceback is more useful.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/97Created database v2 from template is wrong2020-06-18T13:58:35ZAmir MOHAMMADICreated database v2 from template is wrongI have created a v2 database (found in [prefix.tar.xz](/uploads/37186a98f55a1d68e7b316564b8ae85e/prefix.tar.xz))
but when I tried to finally run it from an experiment, I got:
```
$ beat exp run amohammadi/amohammadi/iris-pad-with-train/1...I have created a v2 database (found in [prefix.tar.xz](/uploads/37186a98f55a1d68e7b316564b8ae85e/prefix.tar.xz))
but when I tried to finally run it from an experiment, I got:
```
$ beat exp run amohammadi/amohammadi/iris-pad-with-train/1/iris-pad-with-train
Created cache path `/home/amir/idiap/git/beat/livedet2020-iris-beat-demo/prefix/cache'
Index for database livedet_iris_2020/1 not found, building it
Index for database livedet_iris_2020/1 not found, building it
Running `amohammadi/lbp-feature-extractor/1' for block `test_features'
Start the execution of 'amohammadi/lbp-feature-extractor/1'
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 123, in run
return getattr(obj, method)(*args, **kwargs)
TypeError: setup() got an unexpected keyword argument 'start_index'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/bin/beat", line 11, in <module>
sys.exit(main())
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/decorators.py", line 92, in _decorator
value = view_func(*args, **kwargs)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 780, in run
return run_experiment(config, name, force, docker, local, environment, quiet)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/cmdline/experiments.py", line 278, in run_experiment
result = executor.process()
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/core/execution/local.py", line 360, in process
loop_socket=self.loop_socket,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/execution/algorithm.py", line 166, in __init__
databases=databases,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/helpers.py", line 248, in create_inputs_from_configuration
end_index=end_index,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/database.py", line 184, in setup
end_index=end_index,
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 127, in run
six.reraise(exc, exc(value), traceback)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/home/amir/miniconda/envs/iris/lib/python3.7/site-packages/beat/backend/python/loader.py", line 123, in run
return getattr(obj, method)(*args, **kwargs)
RuntimeError: setup() got an unexpected keyword argument 'start_index'
^CException ignored in: <module 'threading' from '/home/amir/miniconda/envs/iris/lib/python3.7/threading.py'>
Traceback (most recent call last):
File "/home/amir/miniconda/envs/iris/lib/python3.7/threading.py", line 1307, in _shutdown
lock.acquire()
KeyboardInterrupt
```Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/95Schema: missing constraints2020-03-10T16:08:31ZSamuel GAISTSchema: missing constraintsOn the platform, fields like the short descriptions have for example size constraints.
These should be reflected here to avoid issues when creating a new object.On the platform, fields like the short descriptions have for example size constraints.
These should be reflected here to avoid issues when creating a new object.https://gitlab.idiap.ch/beat/beat.core/-/issues/93New environment descovery does not respect the raise on error property2020-01-29T13:15:19ZSamuel GAISTNew environment descovery does not respect the raise on error propertyIf the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call ...If the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call `beat editor refresh-env` on a machine without docker or with the daemon stopped.
A current workaround is to create the `.environments.json` file in the prefix with `{}` as its content.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/92Simplifying toolchain connectivity2019-11-18T09:14:34ZAndré AnjosSimplifying toolchain connectivityThought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our user...Thought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our users a bit: why not avoiding this discrete port to port connection between blocks and accept "whole" block connections only?
The idea is to design toolchains only based on block connections instead of block input/outputs. As a result, a block that receives as an input data from another block, may access any of its outputs.
I'm not sure about the technical challenge(s) behind this, but that would definitely simplify the toolchain editor a bit.
My current understanding is that this does not affect block synchronization and may be relatively "easy" to implement.
@samuel.gaist: what do you think? Easy to roll out?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/91LICENSE file location in conda recipe is wrong2019-11-14T16:07:45ZAndré AnjosLICENSE file location in conda recipe is wrongIt should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.It should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/90Execute experiment in different environment2019-10-07T13:31:09ZSamuel GAISTExecute experiment in different environmentCurrently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of package...Currently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of packages.
While this can't be done with the local executor, it can be implemented using the subprocess executor.
The subprocess executor makes call to the scripts needed to run the algorithms/databases processes as would be with the docker executor. This fact can be used to run the scripts from another environment and thus will use the packages available there.
This will also simplify the creation of new execution environment and their transition do docker images.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/89Improve output error messages2019-10-04T08:44:08ZSamuel GAISTImprove output error messagesWhen not enough data is written on an output, the related cache will be cleaned.
An error message must be shown to the user so he knows what is happening.When not enough data is written on an output, the related cache will be cleaned.
An error message must be shown to the user so he knows what is happening.https://gitlab.idiap.ch/beat/beat.core/-/issues/87Remove parameters from analyzer block2019-08-19T19:15:28ZSamuel GAISTRemove parameters from analyzer blockCurrently analyzer blocks are treated mostly like normal algorithm blocks except the handling of the output.
This means that they can also have parameters. After several discussions, it has been concluded that having the possibility to ...Currently analyzer blocks are treated mostly like normal algorithm blocks except the handling of the output.
This means that they can also have parameters. After several discussions, it has been concluded that having the possibility to parametrize an analyzer would bring an unwelcomed level of complexity especially when creating reports.
If an analyzer should provide different outputs, then it should come in several versions to make it clear from the start what it will provide and how the analysis is done.
The following decisions have been made:
- Keep the current few analyzers that do have parameters as they are
- Remove the parameters field from the V2 algorithm schema for the analyzer definitionSoft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/86Loop parts name2019-07-31T08:30:04ZSamuel GAISTLoop parts nameBased on experience and discussions, the loop parts name are not really clear.
They will be changed to the following to make things clearer:
- XXX_loop_user -> XXX_loop_processor
- XXX_loop -> XXX_loop_evaluator
This will require upda...Based on experience and discussions, the loop parts name are not really clear.
They will be changed to the following to make things clearer:
- XXX_loop_user -> XXX_loop_processor
- XXX_loop -> XXX_loop_evaluator
This will require updates to the corresponding algorithms.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/85Synchronised input for loop blocks2019-07-30T12:56:03ZSamuel GAISTSynchronised input for loop blocksCurrently the two algorithm making a loop macro block are not synchronized input wise.
This is required in sequential mode so the validating part of the loop knows it uses
the same data as the current processing part.
This issue tracks...Currently the two algorithm making a loop macro block are not synchronized input wise.
This is required in sequential mode so the validating part of the loop knows it uses
the same data as the current processing part.
This issue tracks that input synchronization.Soft loopsSamuel GAISTSamuel GAIST