beat.core issueshttps://gitlab.idiap.ch/beat/beat.core/-/issues2015-03-06T08:04:27Zhttps://gitlab.idiap.ch/beat/beat.core/-/issues/3Repeated connection is not detected on toolchain JSON (migrated from github)2015-03-06T08:04:27ZAndré AnjosRepeated connection is not detected on toolchain JSON (migrated from github)By mistake, I created a repeated connection from one block to another, on my list of connections. The platform has signaled the toolchain to be valid, but when running, the following error occurred:
```python
File "/remote/filer.gx...By mistake, I created a repeated connection from one block to another, on my list of connections. The platform has signaled the toolchain to be valid, but when running, the following error occurred:
```python
File "/remote/filer.gx/user.active/aanjos/work/projects/beat/beat/eggs/ipdb-0.7-py2.7.egg/ipdb/__main__.py", line 138, in main
pdb._runscript(mainpyfile)
File "/usr/lib/python2.7/pdb.py", line 1233, in _runscript
self.run(statement)
File "/usr/lib/python2.7/bdb.py", line 400, in run
exec cmd in globals, locals
File "<string>", line 1, in <module>
File "./bin/run_toolchain.py", line 3, in <module>
import sys
File "/remote/filer.gx/user.active/aanjos/work/projects/beat/beat/src/beat.core/beat/core/scripts/run_toolchain.py", line 186, in main
if isinstance(connection.output.block, Database):
AttributeError: 'NoneType' object has no attribute 'output'
Uncaught exception. Entering post mortem debugging
```
I guess that should be relatively easy to fix.Tutorial for the FG2015 ConferenceAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/2Warning for unconnected/multiply connected blocks (migrated from github)2015-03-06T08:05:11ZAndré AnjosWarning for unconnected/multiply connected blocks (migrated from github)Along the same lines as #39, it would be convenient that both `check_toolchain.py` and `run_toolchain.py` would check for unconnected blocks and issue a **warning** in case one is detected. It would be an immense helper for debugging new...Along the same lines as #39, it would be convenient that both `check_toolchain.py` and `run_toolchain.py` would check for unconnected blocks and issue a **warning** in case one is detected. It would be an immense helper for debugging new toolchains.Tutorial for the FG2015 ConferenceAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/35Move command-line client to another package2017-08-06T11:17:03ZAndré AnjosMove command-line client to another packageAs discussed, we should move the command-line client to another package, since it depends on the `beat.web` API. This creates some sort of reverse dependence which we should get rid of.As discussed, we should move the command-line client to another package, since it depends on the `beat.web` API. This creates some sort of reverse dependence which we should get rid of.BEAT TourAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/32Dash '-' as a valid (or invalid) character for element names (dataformats, al...2017-08-06T11:17:03ZLaurent EL SHAFEYDash '-' as a valid (or invalid) character for element names (dataformats, algorithms, etc.)?I've noticed an inconsistency wrt. the usage of the dash '-' character for element names.
There is currently a bunch of elements (e.g. algorithm 'ptome/fingervein-miura_scoring/1/') defined on the platform, which contains the '-' char...I've noticed an inconsistency wrt. the usage of the dash '-' character for element names.
There is currently a bunch of elements (e.g. algorithm 'ptome/fingervein-miura_scoring/1/') defined on the platform, which contains the '-' character.
In contrast, the '-' character is not 'always' part of the valid character at the JSON validation schema level (see beat/core/schema/experiment.json line 48).
The list of valid characters needs to be properly defined, and the characters accepted by the beat.web interface need to match the ones of the JSON validation schema.BEAT TourLaurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.core/-/issues/26Documentation updates2017-08-06T11:17:04ZAndré AnjosDocumentation updatesThe API for beat.core is rather complete and well-documented. It would be good to catch up with the rest of the documentation before our next deliverable (by end of April/2015).The API for beat.core is rather complete and well-documented. It would be good to catch up with the rest of the documentation before our next deliverable (by end of April/2015).BEAT TourLaurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.core/-/issues/44Agent is unprotected in case the user sends to many "nxt" commands2017-08-06T11:17:03ZAndré AnjosAgent is unprotected in case the user sends to many "nxt" commandsWe should put in place:
1. A check in the agent side (around line 150), that prevents the agent to measure the length of `None'
2. A new protocol reply possibility for an erroneous condition on the `nxt` operation.
We should put in place:
1. A check in the agent side (around line 150), that prevents the agent to measure the length of `None'
2. A new protocol reply possibility for an erroneous condition on the `nxt` operation.
BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/42[io] Execute externally currently requires twisted (async.run)2017-08-06T11:17:03ZLaurent EL SHAFEY[io] Execute externally currently requires twisted (async.run)With the introduction of the new I/O framework, the execution of a job requires to launch the I/O daemon (that will launch and control the user process). This daemon currently needs twisted (async.run) to be executed, while we would idea...With the introduction of the new I/O framework, the execution of a job requires to launch the I/O daemon (that will launch and control the user process). This daemon currently needs twisted (async.run) to be executed, while we would ideally prefer not to have such a dependency in beat.core.
To fix this, an optimal decapsulation of the communication via pipes from the I/O daemon would be required, as well as its integration into beat.core.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/41[io] Optimization of I/O when using pipes2017-08-06T11:17:03ZLaurent EL SHAFEY[io] Optimization of I/O when using pipesWith the introduction of communication via pipes between the user process and the daemon, it may happen that a chunk of data is decoded from a baseformat and then re-encoded (into a baseformat) to send it through pipe. This is the case, ...With the introduction of communication via pipes between the user process and the daemon, it may happen that a chunk of data is decoded from a baseformat and then re-encoded (into a baseformat) to send it through pipe. This is the case, when next() is called: a chunk of data is then read using a CachedDataSource (that will decode from a baseformat) and then sent through a pipe (and hence will be re-encoded into a baseformat). This is of course suboptimal and may be fixed by specializing the CachedDataSource.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/40Cleanup of executor/dataformats/algorithms/libraries/schema loading2017-08-06T11:17:03ZAndré AnjosCleanup of executor/dataformats/algorithms/libraries/schema loadingAs of today, we moved the definition of these objects and functions to beat.backend.python. The backend does not need to be able to validate objects, only to load them and instantiate a runner. We should revise and clean-up this relation...As of today, we moved the definition of these objects and functions to beat.backend.python. The backend does not need to be able to validate objects, only to load them and instantiate a runner. We should revise and clean-up this relationship.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/39inputs/outputs cleanup w.r.t beat.backend.python2017-08-06T11:17:03ZAndré Anjosinputs/outputs cleanup w.r.t beat.backend.pythonAs of today there is some functionality of beat.backend.python that is duplicated in this package. We should clean-up and organised it so there is no repetition of code, respecting a good DRY principle.As of today there is some functionality of beat.backend.python that is duplicated in this package. We should clean-up and organised it so there is no repetition of code, respecting a good DRY principle.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/38discover_environments() does not crash when the same environment/version is d...2017-08-06T11:17:03ZLaurent EL SHAFEYdiscover_environments() does not crash when the same environment/version is detected multiple timesWhile this should raise an exception.While this should raise an exception.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/34JSON declarations should be versioned2017-08-06T11:17:03ZAndré AnjosJSON declarations should be versionedWe should include a version field on the JSON declaration to allow for changes on the declaration to co-exist.We should include a version field on the JSON declaration to allow for changes on the declaration to co-exist.BTAS TutorialAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/101Add tmpfs when running a container2020-07-29T09:18:53ZSamuel GAISTAdd tmpfs when running a containerDespite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file c...Despite the tests run in the various aspects of the BEAT code base which showed that having the container start read-only was successful, actual run on the platform started to unsuccessfully fail with an error related to temporary file creation coming from the entrypoint script loading the correct conda environment.
"Unsuccessfully failed" here means that the full experiment run was marked as good despite that error happening which in fact did not even allow for a proper start of the execution procedure.
The fix for this situation is to add a minimal writable tmpfs for /tmp in order to have the space necessary for that file to be created.
It might also be useful to have one for /run even if it does not look like it is being used currently (at least in the same way as /tmp seemed to not be used).Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/93New environment descovery does not respect the raise on error property2020-01-29T13:15:19ZSamuel GAISTNew environment descovery does not respect the raise on error propertyIf the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call ...If the discovery process using image labels is run on a machine without docker running or if the daemon is not running, it will fail as expected however the error handling does not follow the raise on error property.
To reproduce, call `beat editor refresh-env` on a machine without docker or with the daemon stopped.
A current workaround is to create the `.environments.json` file in the prefix with `{}` as its content.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/90Execute experiment in different environment2019-10-07T13:31:09ZSamuel GAISTExecute experiment in different environmentCurrently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of package...Currently, when executing an experiment on a local machine, the current environment is used and therefor must contain all the packages needed.
This is sub-optimal as several different experiences might required different sets of packages.
While this can't be done with the local executor, it can be implemented using the subprocess executor.
The subprocess executor makes call to the scripts needed to run the algorithms/databases processes as would be with the docker executor. This fact can be used to run the scripts from another environment and thus will use the packages available there.
This will also simplify the creation of new execution environment and their transition do docker images.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/87Remove parameters from analyzer block2019-08-19T19:15:28ZSamuel GAISTRemove parameters from analyzer blockCurrently analyzer blocks are treated mostly like normal algorithm blocks except the handling of the output.
This means that they can also have parameters. After several discussions, it has been concluded that having the possibility to ...Currently analyzer blocks are treated mostly like normal algorithm blocks except the handling of the output.
This means that they can also have parameters. After several discussions, it has been concluded that having the possibility to parametrize an analyzer would bring an unwelcomed level of complexity especially when creating reports.
If an analyzer should provide different outputs, then it should come in several versions to make it clear from the start what it will provide and how the analysis is done.
The following decisions have been made:
- Keep the current few analyzers that do have parameters as they are
- Remove the parameters field from the V2 algorithm schema for the analyzer definitionSoft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/86Loop parts name2019-07-31T08:30:04ZSamuel GAISTLoop parts nameBased on experience and discussions, the loop parts name are not really clear.
They will be changed to the following to make things clearer:
- XXX_loop_user -> XXX_loop_processor
- XXX_loop -> XXX_loop_evaluator
This will require upda...Based on experience and discussions, the loop parts name are not really clear.
They will be changed to the following to make things clearer:
- XXX_loop_user -> XXX_loop_processor
- XXX_loop -> XXX_loop_evaluator
This will require updates to the corresponding algorithms.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/85Synchronised input for loop blocks2019-07-30T12:56:03ZSamuel GAISTSynchronised input for loop blocksCurrently the two algorithm making a loop macro block are not synchronized input wise.
This is required in sequential mode so the validating part of the loop knows it uses
the same data as the current processing part.
This issue tracks...Currently the two algorithm making a loop macro block are not synchronized input wise.
This is required in sequential mode so the validating part of the loop knows it uses
the same data as the current processing part.
This issue tracks that input synchronization.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/84Implement output for loop block2019-07-30T07:15:14ZSamuel GAISTImplement output for loop blockCurrently the loop blocks don't have any outputs.
This issue tracks the implementation of this feature in the loop block.
One requested constraint for these outputs is that the write to them should happen when a write occurs in the out...Currently the loop blocks don't have any outputs.
This issue tracks the implementation of this feature in the loop block.
One requested constraint for these outputs is that the write to them should happen when a write occurs in the output of a loop user block.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/82Improve duplicate key handling2019-06-19T05:13:48ZSamuel GAISTImprove duplicate key handlingThis issue tracks the implementation of duplicated key handling in JSON files used.
In normal situation, the files should be edited and stored using beat/beat.editor> or currently still beat/beat.web> however if one does edit the declar...This issue tracks the implementation of duplicated key handling in JSON files used.
In normal situation, the files should be edited and stored using beat/beat.editor> or currently still beat/beat.web> however if one does edit the declaration file by hand, errors like duplicated key might be introduced which won't be caught as loading a JSON file line by line will result in a dictionary with the last entry being used (which is normal behavior when dealing with dictionary type of containers).
In our case, the system will generate an error so that nothing will run with unexpected values.Soft loopsSamuel GAISTSamuel GAIST