beat issueshttps://gitlab.idiap.ch/groups/beat/-/issues2020-06-29T12:45:33Zhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/34The database View class could be simplified2020-06-29T12:45:33ZAmir MOHAMMADIThe database View class could be simplifiedI don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just...I don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just a container as far as I can see. So, I wonder if having something like:
```python
class Train:
"""The training set"""
def __init__(self, parameters=None, root_folder=None):
self.parameters = parameters
self.root_folder = root_folder
# initialize the db interface here
# e.g. load a csv file
import pandas as pd
self.df = pd.read_csv(...)
def __len__(self):
# return the total number of items in here
return len(self.df)
def __getitem__(self, index):
# return the nth row in here
row = self.df.iloc[index]
# load the data here if necessary
row.image = load(row.filename)
return row
```
would not satisfy the requirements.Interactivity and Intuitive APIhttps://gitlab.idiap.ch/beat/docs/-/issues/15Analyzers work differently from algorithms but there is no docs on them.2020-06-29T12:44:33ZAmir MOHAMMADIAnalyzers work differently from algorithms but there is no docs on them.The syntax of algorithms are different from analyzers but I could not find documentation on this.
For example, an algorithm works like this:
```python
def process(self, inputs, data_loaders, outputs):
outputs["features"].wr...The syntax of algorithms are different from analyzers but I could not find documentation on this.
For example, an algorithm works like this:
```python
def process(self, inputs, data_loaders, outputs):
outputs["features"].write({"value": out})
```
while an analyzer would be:
```python
def process(self, inputs, data_loaders, output):
output.write({"features": {"value": out}})
```
I have two issues here:
1. why analyzers code should be different?
2. where is this documented?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.editor/-/issues/196rearrange the inputs in a toolchain block2020-06-16T16:03:32ZZohreh MOSTAANIrearrange the inputs in a toolchain blockIt would be very useful if the user is able to reorder the inputs when they are defined in a block. Sometimes you may just make a block with the inputs but it will be easier to connect them to other blocks if they were in another order. ...It would be very useful if the user is able to reorder the inputs when they are defined in a block. Sometimes you may just make a block with the inputs but it will be easier to connect them to other blocks if they were in another order. This is different from renaming though.[v2] 2 - Toolchain editorhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/33Add multi-processing support to RemoteDataSource2020-06-08T14:00:56ZSamuel GAISTAdd multi-processing support to RemoteDataSourceFollowing #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, th...Following #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, this much be changed to something different. Essentially multiple clients to one server in request/response mode.https://gitlab.idiap.ch/beat/beat.cmdline/-/issues/69Cannot create a new database2020-06-02T14:14:34ZAmir MOHAMMADICannot create a new databaseRunning `beat databases create myiris/1` will result in:
```
Traceback (most recent call last):
File "/home/amir/miniconda/envs/beat-tutorial/bin/beat", line 11, in <module>
sys.exit(main())
File "/home/amir/miniconda/envs/beat-t...Running `beat databases create myiris/1` will result in:
```
Traceback (most recent call last):
File "/home/amir/miniconda/envs/beat-tutorial/bin/beat", line 11, in <module>
sys.exit(main())
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/cmdline/click_helper.py", line 121, in invoke
return super().invoke(ctx)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/cmdline/decorators.py", line 92, in _decorator
value = view_func(*args, **kwargs)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/cmdline/commands.py", line 162, in create_impl
return common.create(config.path, asset_info.asset_type, names)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/cmdline/common.py", line 1009, in create
obj = TYPE_VALIDATOR[asset_type](prefix, data=None) # the default object
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/core/database.py", line 131, in __init__
super(Database, self).__init__(prefix, data, dataformat_cache)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/backend/python/database.py", line 264, in __init__
self._load(name, dataformat_cache)
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/core/database.py", line 167, in _load
self.prefix
File "/home/amir/miniconda/envs/beat-tutorial/lib/python3.7/site-packages/beat/core/database.py", line 69, in get_first_procotol_template
raise RuntimeError("Invalid prefix content, no protocol template available")
RuntimeError: Invalid prefix content, no protocol template available
```Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/31Support bytes python type2020-05-27T16:23:17ZSamuel GAISTSupport bytes python typeCurrently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end ...Currently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end load it back from string.
Example of working code:
Output:
```python
obj = dict(field1=data.value*2, field2=data.value)
dumped = pickle.dumps(obj) # binary
encoded = base64.b64encode(dumped) # base64 bytes
string = encoded.decode("ascii") # "stringified"
outputs["out_data"].write({"value": string})
```
**WARNING** Do **not** call str(encoded), it won't be a "real string":
```python
example = b"whatever"
example_str = str(example)
example_str
# output is "b'whatever'"
```
Input:
```python
in_data = inputs["in_data"].data # string data
decoded = base64.b64decode(in_data.value) # decode to bytes
obj = pickle.loads(decoded) # obj is a dict and can be read
```
This issue tracks the implementation of the support for the python bytes type.https://gitlab.idiap.ch/beat/beat.cmdline/-/issues/68beat exp fork command is cumbersome to use2020-05-15T12:56:46ZAmir MOHAMMADIbeat exp fork command is cumbersome to useWhen you fork an experiment in the command line, you have to something like:
```sh
$ beat exp fork test/test/iris/1/iris amir/test/iris/1/means
```
but you can only change the username and name of the experiment when you fork.
When usin...When you fork an experiment in the command line, you have to something like:
```sh
$ beat exp fork test/test/iris/1/iris amir/test/iris/1/means
```
but you can only change the username and name of the experiment when you fork.
When using beat.editor 2.0.0, when I fork an experiment, it only asks for the exp name.
Beat.editor somehow decided to put the forked experiment under user `amir` but I don't know
yet how it decided on that.
Shouldn't this command be also similar?
Something like:
```sh
$ beat exp fork test/test/iris/1/iris means
$ beat exp fork --user amir test/test/iris/1/iris means
```https://gitlab.idiap.ch/beat/beat.core/-/issues/29Loops between blocks (MAP/Reduce)2020-05-13T06:32:34ZTiago de Freitas PereiraLoops between blocks (MAP/Reduce)For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And ...For some iterative algorithms we can create procedures to parallelize them.
This parallelization involves a Map step (when you slice the data between the nodes) and the Reduce step (when you reduce/gather some data from the node).
And you do this (Map/Reduce) several times.
To implement that using platform, you need two blocks, one for the Map and one for the Reduce.
Since the flow of the data in the platform is just forward, it is not possible to do a loop between these two blocks
Would be nice to have that feature.https://gitlab.idiap.ch/beat/beat.editor/-/issues/217Update editor needing complex type support2020-02-12T13:15:12ZSamuel GAISTUpdate editor needing complex type supportThe latest version of beat/beat.core> following beat/beat.core!65 doesn't contain complex types anymore as they are specific to the Dataformat asset.
The DataformatEditor must be updated to handle that change.The latest version of beat/beat.core> following beat/beat.core!65 doesn't contain complex types anymore as they are specific to the Dataformat asset.
The DataformatEditor must be updated to handle that change.[v2] 1 - Edition/Visualization for small editorshttps://gitlab.idiap.ch/beat/beat.editor/-/issues/216Update basetype getter2020-02-12T13:14:31ZSamuel GAISTUpdate basetype getterbeat/beat.core!65 modifies heavily the schema to be more accurate.
The basetype definition changes in that regards and the current setter doesn't support it yet.beat/beat.core!65 modifies heavily the schema to be more accurate.
The basetype definition changes in that regards and the current setter doesn't support it yet.[v2] 1 - Edition/Visualization for small editorsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.editor/-/issues/193breaking connections in a toolchain to by pass the blocks2020-02-12T12:52:21ZZohreh MOSTAANIbreaking connections in a toolchain to by pass the blocksI think it would be good if the connections in a toolchain could break at some points and bypass blocks instead of directly connecting inputs and outputs. Something like what is available on the beat platform. The reason is that even tho...I think it would be good if the connections in a toolchain could break at some points and bypass blocks instead of directly connecting inputs and outputs. Something like what is available on the beat platform. The reason is that even though it is more convenient while making a toolchain to just connect an output to an input it is very difficult to understand what is going on when you look at an experiment using that toolchain, specially when the experiments get bigger and the connections are crossing each other. Even if there was a mechanism to highlight a chosen connection in the experiment page can be really helpful. This mechanism is already there when making a toolchain but not when you look at the experiment.[v2] 2 - Toolchain editorhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/228Feature request: Be able to have multi window open2020-02-12T12:42:39ZZohreh MOSTAANIFeature request: Be able to have multi window openIt would be nice if you are able to have multiple assets open at the same time, since sometimes you may need it.
There are to options here:
1. Having multiple windows open in the same instance of editor. Kind of just splitting the edit...It would be nice if you are able to have multiple assets open at the same time, since sometimes you may need it.
There are to options here:
1. Having multiple windows open in the same instance of editor. Kind of just splitting the editor window.
2. Having a new instance of editor opened from terminal. In this case there might be complications with the log info shown in the terminal. The question is are they useful for user or not?https://gitlab.idiap.ch/beat/beat.core/-/issues/92Simplifying toolchain connectivity2019-11-18T09:14:34ZAndré AnjosSimplifying toolchain connectivityThought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our user...Thought about dumping this here, for tracking.
While discussing with @flavio.tarsetti on the toolchain editor and @tiago.pereira on snakemake and workflow definitions, a thought occurred that may simplify our lives and those of our users a bit: why not avoiding this discrete port to port connection between blocks and accept "whole" block connections only?
The idea is to design toolchains only based on block connections instead of block input/outputs. As a result, a block that receives as an input data from another block, may access any of its outputs.
I'm not sure about the technical challenge(s) behind this, but that would definitely simplify the toolchain editor a bit.
My current understanding is that this does not affect block synchronization and may be relatively "easy" to implement.
@samuel.gaist: what do you think? Easy to roll out?Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.web/-/issues/536Some of the Models defined in django are not well documented2019-09-05T11:36:57ZZohreh MOSTAANISome of the Models defined in django are not well documentedThe "swagger.yaml" file that includes the beat.web api information that can be rendered into the sphinx documentation does not include complete information for all the models included in a response object. Some of the basic ones are show...The "swagger.yaml" file that includes the beat.web api information that can be rendered into the sphinx documentation does not include complete information for all the models included in a response object. Some of the basic ones are shown though which means that the package produces this yaml file is working as expected.
It seems that those objects which are not shown, are not well documented in the code and because there are several layers of inheritance for each class, it is difficult to find where exactly the problem lies.
With a brief discussion with @samuel.gaist we tried something fast but it did not work. It required a deeper dive into the implementation.https://gitlab.idiap.ch/beat/beat.web/-/issues/533Strange behaviour concerning a POST method that needed to be GET.2019-08-22T15:38:08ZZohreh MOSTAANIStrange behaviour concerning a POST method that needed to be GET.In the `api.py` file for `report`, there are two classes that have POST methods which in fact should have been GET.
`ReportRSTCompileView` and `ReportRSTCompileAnonView`. The definitions are here:
https://gitlab.idiap.ch/beat/beat.web/b...In the `api.py` file for `report`, there are two classes that have POST methods which in fact should have been GET.
`ReportRSTCompileView` and `ReportRSTCompileAnonView`. The definitions are here:
https://gitlab.idiap.ch/beat/beat.web/blob/master/beat/web/reports/api.py
Both of the mentioned classes are using another method `compileTextItem` defined in `models.py`. This function is not readable and it raises an exception error `"Error in ReportRSTCompileView: The Python docutils library isn't installed."` which does not seem to belong here. The definition is here:
https://gitlab.idiap.ch/beat/beat.web/blob/master/beat/web/reports/models.pyhttps://gitlab.idiap.ch/beat/beat.web/-/issues/247[reports] Workflow for adding a Table of results is not very intuitive2019-07-04T08:37:03ZAndré Anjos[reports] Workflow for adding a Table of results is not very intuitiveThe current workflow to add a Table of results gets the user through a sequence of clicks up to the point a list with select boxes is shown.
At this point, there is no "set" button or anything that allows the user to dismiss/apply the...The current workflow to add a Table of results gets the user through a sequence of clicks up to the point a list with select boxes is shown.
At this point, there is no "set" button or anything that allows the user to dismiss/apply the options. It is not very automatic that clicking outside the pop-up window makes it apply the options. I'd rather think it should cancel the action...
Anyways, some improvement on this area would be welcome.https://gitlab.idiap.ch/beat/beat.web/-/issues/237Formalize plotter API2019-07-04T08:37:03ZAndré AnjosFormalize plotter APILike `algorithms`, we should make sure we have a formal API to handle `plotters`. Only the `system` user may upload/change objects. All users may download and list `plotters` which are accessible to them.Like `algorithms`, we should make sure we have a formal API to handle `plotters`. Only the `system` user may upload/change objects. All users may download and list `plotters` which are accessible to them.https://gitlab.idiap.ch/beat/beat.web/-/issues/220User object privacy (encryption)2019-07-04T08:37:03ZAndré AnjosUser object privacy (encryption)As discussed in various occasions, user objects at the platform, despite not accessible anymore by admin/staff users via the web GUI, are still accessible on disk by administrators.
We must setup a scheme to get rid of this "bug" as t...As discussed in various occasions, user objects at the platform, despite not accessible anymore by admin/staff users via the web GUI, are still accessible on disk by administrators.
We must setup a scheme to get rid of this "bug" as the trust on the platform itself depends on how well we can keep our users' information.
Here is a suggestion to get the ball rolling:
1. Stored objects have declaration, code and documentation (files) encrypted using the user's password
2. At execution, the complete experiment tree is passed to the scheduler, unencrypted, which relies solely on it (and not any longer on information on disk) to execute the experiment.
Possible issues:
1. How to deal with shared objects (public objects should be left unencrypted, I guess, but privately shared ones are not that clear)? Maybe we'll need to duplicate objects at sharing time.
2. How to effectively implement this?
Side effects:
1. The scheduler would have no longer to verify the hash of submitted experiments.https://gitlab.idiap.ch/beat/beat.web/-/issues/178Search: Column sorting and reordering directly from the table header2019-07-04T08:37:03ZPhilip ABBETSearch: Column sorting and reordering directly from the table headerWith support for multi-column reordering.
Include a replacement of the custom SQL part by a more flexible Javascript implementation.With support for multi-column reordering.
Include a replacement of the custom SQL part by a more flexible Javascript implementation.Philip ABBETPhilip ABBEThttps://gitlab.idiap.ch/beat/beat.web/-/issues/177Search: Add one filter: algorithm in a given block2019-07-04T08:37:03ZPhilip ABBETSearch: Add one filter: algorithm in a given blockPhilip ABBETPhilip ABBET