beat.backend.python issueshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues2020-06-29T12:45:33Zhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/34The database View class could be simplified2020-06-29T12:45:33ZAmir MOHAMMADIThe database View class could be simplifiedI don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just...I don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just a container as far as I can see. So, I wonder if having something like:
```python
class Train:
"""The training set"""
def __init__(self, parameters=None, root_folder=None):
self.parameters = parameters
self.root_folder = root_folder
# initialize the db interface here
# e.g. load a csv file
import pandas as pd
self.df = pd.read_csv(...)
def __len__(self):
# return the total number of items in here
return len(self.df)
def __getitem__(self, index):
# return the nth row in here
row = self.df.iloc[index]
# load the data here if necessary
row.image = load(row.filename)
return row
```
would not satisfy the requirements.Interactivity and Intuitive APIhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/36Editing the json file of database views does not invalidate the cache.2020-07-28T09:45:09ZAmir MOHAMMADIEditing the json file of database views does not invalidate the cache.Steps to reproduce:
```
$ beat cache clear
$ beat exp pull amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
Index for database atnt/5 not found, building it
li...Steps to reproduce:
```
$ beat cache clear
$ beat exp pull amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
Index for database atnt/5 not found, building it
lib/python3.7/site-packages/setuptools/distutils_patch.py:26: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
"Distutils was imported before Setuptools. This usage is discouraged "
Index for database atnt/5 not found, building it
Index for database atnt/5 not found, building it
Running `amohammadi/pca/1' for block `linear_machine_training'
Start the execution of 'amohammadi/pca/1'
Block did not execute properly - outputs were reset
Standard output:
Standard error:
Captured user error:
File "lib/python3.7/site-packages/bob/io/base/__init__.py", line 143, in load
return File(inputs, 'r').read()
RuntimeError: File - constructor: C++ exception caught: 'file '/path_to_db_folder/att_faces/s1/1.pgm' is not readable'
Captured system error:
Error: Error occured: returned value is 1
Removing cache files: No data written
```
* change the json file `prefix/databases/atnt/5.json` of the database and fix the the `root_folder`
* run the experiment again:
```
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
RuntimeError: File - constructor: C++ exception caught: 'file '/path_to_db_folder/att_faces/s1/1.pgm' is not readable'
```Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/35BEAT componenets are tied to a prefix2022-03-03T17:35:06ZAmir MOHAMMADIBEAT componenets are tied to a prefixThroughout all code and components of BEAT, a prefix is required and this requirement makes it impossible to define and run BEAT experiments interactively.
Here is a tentative plan for refactoring the code:
1. [ ] Update BEAT compone...Throughout all code and components of BEAT, a prefix is required and this requirement makes it impossible to define and run BEAT experiments interactively.
Here is a tentative plan for refactoring the code:
1. [ ] Update BEAT component classes so that they can be created on the fly without pointing to a prefix
2. [ ] Implement a global config object to keep track of user's config such as where the prefix is or what the username is. This will help users provide less information when creating objects on the fly.
3. [ ] Dynamic creation of experiment/toolchain with running the python code in a kind of graph mode. This will be similar to how graphs are constructed in Python using tensorflow or dask.
3. [ ] We would also need a singleton class to hold the prefix objects in memory to avoid passing around caches.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/33Add multi-processing support to RemoteDataSource2020-06-08T14:00:56ZSamuel GAISTAdd multi-processing support to RemoteDataSourceFollowing #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, th...Following #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, this much be changed to something different. Essentially multiple clients to one server in request/response mode.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/31Support bytes python type2020-05-27T16:23:17ZSamuel GAISTSupport bytes python typeCurrently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end ...Currently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end load it back from string.
Example of working code:
Output:
```python
obj = dict(field1=data.value*2, field2=data.value)
dumped = pickle.dumps(obj) # binary
encoded = base64.b64encode(dumped) # base64 bytes
string = encoded.decode("ascii") # "stringified"
outputs["out_data"].write({"value": string})
```
**WARNING** Do **not** call str(encoded), it won't be a "real string":
```python
example = b"whatever"
example_str = str(example)
example_str
# output is "b'whatever'"
```
Input:
```python
in_data = inputs["in_data"].data # string data
decoded = base64.b64decode(in_data.value) # decode to bytes
obj = pickle.loads(decoded) # obj is a dict and can be read
```
This issue tracks the implementation of the support for the python bytes type.