beat.backend.python issueshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues2020-07-28T09:45:09Zhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/36Editing the json file of database views does not invalidate the cache.2020-07-28T09:45:09ZAmir MOHAMMADIEditing the json file of database views does not invalidate the cache.Steps to reproduce:
```
$ beat cache clear
$ beat exp pull amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
Index for database atnt/5 not found, building it
li...Steps to reproduce:
```
$ beat cache clear
$ beat exp pull amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
Index for database atnt/5 not found, building it
lib/python3.7/site-packages/setuptools/distutils_patch.py:26: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
"Distutils was imported before Setuptools. This usage is discouraged "
Index for database atnt/5 not found, building it
Index for database atnt/5 not found, building it
Running `amohammadi/pca/1' for block `linear_machine_training'
Start the execution of 'amohammadi/pca/1'
Block did not execute properly - outputs were reset
Standard output:
Standard error:
Captured user error:
File "lib/python3.7/site-packages/bob/io/base/__init__.py", line 143, in load
return File(inputs, 'r').read()
RuntimeError: File - constructor: C++ exception caught: 'file '/path_to_db_folder/att_faces/s1/1.pgm' is not readable'
Captured system error:
Error: Error occured: returned value is 1
Removing cache files: No data written
```
* change the json file `prefix/databases/atnt/5.json` of the database and fix the the `root_folder`
* run the experiment again:
```
$ beat exp run amohammadi/tutorial/eigenface/1/atnt-eigenfaces-67-comp
RuntimeError: File - constructor: C++ exception caught: 'file '/path_to_db_folder/att_faces/s1/1.pgm' is not readable'
```Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/35BEAT componenets are tied to a prefix2022-03-03T17:35:06ZAmir MOHAMMADIBEAT componenets are tied to a prefixThroughout all code and components of BEAT, a prefix is required and this requirement makes it impossible to define and run BEAT experiments interactively.
Here is a tentative plan for refactoring the code:
1. [ ] Update BEAT compone...Throughout all code and components of BEAT, a prefix is required and this requirement makes it impossible to define and run BEAT experiments interactively.
Here is a tentative plan for refactoring the code:
1. [ ] Update BEAT component classes so that they can be created on the fly without pointing to a prefix
2. [ ] Implement a global config object to keep track of user's config such as where the prefix is or what the username is. This will help users provide less information when creating objects on the fly.
3. [ ] Dynamic creation of experiment/toolchain with running the python code in a kind of graph mode. This will be similar to how graphs are constructed in Python using tensorflow or dask.
3. [ ] We would also need a singleton class to hold the prefix objects in memory to avoid passing around caches.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/34The database View class could be simplified2020-06-29T12:45:33ZAmir MOHAMMADIThe database View class could be simplifiedI don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just...I don't know exactly the internals of beat but I wonder why the database view class needs to be this complicated?
There may be some technical reasons behind this but I don't think this should be exposed to the users.
These Views are just a container as far as I can see. So, I wonder if having something like:
```python
class Train:
"""The training set"""
def __init__(self, parameters=None, root_folder=None):
self.parameters = parameters
self.root_folder = root_folder
# initialize the db interface here
# e.g. load a csv file
import pandas as pd
self.df = pd.read_csv(...)
def __len__(self):
# return the total number of items in here
return len(self.df)
def __getitem__(self, index):
# return the nth row in here
row = self.df.iloc[index]
# load the data here if necessary
row.image = load(row.filename)
return row
```
would not satisfy the requirements.Interactivity and Intuitive APIhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/33Add multi-processing support to RemoteDataSource2020-06-08T14:00:56ZSamuel GAISTAdd multi-processing support to RemoteDataSourceFollowing #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, th...Following #32
The current implementation of the processing block to database container uses a one to one connection through ZMQ exclusive pair.
In order to also allow people to use multiprocessing with a dataset as input to a block, this much be changed to something different. Essentially multiple clients to one server in request/response mode.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/32Multiprocessing support for data sources2020-06-08T14:00:57ZAndré AnjosMultiprocessing support for data sourcesAs discussed in today's debugging session with @samuel.gaist and @amohammadi, using a `DataLoader` object in a multiprocessing context is hard:
1. Typically, the underlying `DataSource`'s `fileobj`'s are opened by the time the process i...As discussed in today's debugging session with @samuel.gaist and @amohammadi, using a `DataLoader` object in a multiprocessing context is hard:
1. Typically, the underlying `DataSource`'s `fileobj`'s are opened by the time the process is forked
2. Deep copying the object (which goes through pickling and unpickling it) does not properly reset underlying `fileobj` pointers, which makes multiple processes access the same underlying OS-level file handler, causing unwanted behaviour.
To sort this out, we discussed 2 possible additions to this package:
1. `DataLoader` should have a `reset()` method that resets all underlying `DataSource` opened files, so that they can be correctly copied across multiple processes (e.g. in the event of a `fork()`). It should be relatively easy to do a `reset()` operation across all inputs of a user algorithm, to ensure all data sources are properly reset before an eventual user-guided `fork()`.
2. The underlying `DataSource` should have its pickle/unpickle behaviour patched (via overwriting the `__setstate__` slot of `DataSource`, see reference below), so that unpickling a data source (e.g. indirectly via a data loader deep copy), will call `self.reset()` after its state is unpickled. This would allow a `DataLoader` object to be sent over current mechanisms for inter-process communication (e.g. MPI or `multiprocessing.Queue`), transparently.
References:
* Python fileobj handling: https://stackoverflow.com/questions/1834556/does-a-file-object-automatically-close-when-its-reference-count-hits-zero
* Pickle user guide (see in particular `__getstate__` and `__setstate__` on how to overwrite the pickle/unpickle actions): https://docs.python.org/3/library/pickle.html#object.__getstate__
* On sharing (opened) file pointers in a POSIX system after a `fork()` is issued: https://stackoverflow.com/questions/33899548/file-pointers-after-returning-from-a-forked-child-processSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/31Support bytes python type2020-05-27T16:23:17ZSamuel GAISTSupport bytes python typeCurrently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end ...Currently there's no way to send bytes directly from one algorithm to another.
The current "workaround" is to build the bytes output then use base64 encoding, use a string based type to write the data to the output and on the other end load it back from string.
Example of working code:
Output:
```python
obj = dict(field1=data.value*2, field2=data.value)
dumped = pickle.dumps(obj) # binary
encoded = base64.b64encode(dumped) # base64 bytes
string = encoded.decode("ascii") # "stringified"
outputs["out_data"].write({"value": string})
```
**WARNING** Do **not** call str(encoded), it won't be a "real string":
```python
example = b"whatever"
example_str = str(example)
example_str
# output is "b'whatever'"
```
Input:
```python
in_data = inputs["in_data"].data # string data
decoded = base64.b64decode(in_data.value) # decode to bytes
obj = pickle.loads(decoded) # obj is a dict and can be read
```
This issue tracks the implementation of the support for the python bytes type.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/30Error when waiting on a loop that wasn't started2020-02-06T17:36:09ZSamuel GAISTError when waiting on a loop that wasn't startedIf for some reason the loop, or database, `process` method was not called and `wait` is called, a runtime error will occur because the message handler was not started as expected by the current code.If for some reason the loop, or database, `process` method was not called and `wait` is called, a runtime error will occur because the message handler was not started as expected by the current code.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/28License file location is wrong in conda recipe2019-11-14T15:58:26ZAndré AnjosLicense file location is wrong in conda recipeIt should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.It should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/27Improve loop evaluator synchronized write2019-10-01T10:08:15ZSamuel GAISTImprove loop evaluator synchronized writeThe loop evaluator write method is call once per output write. However, a processor block may have several outputs which means that the evaluator write method will be called as many times as an output is written. So in the case of a sequ...The loop evaluator write method is call once per output write. However, a processor block may have several outputs which means that the evaluator write method will be called as many times as an output is written. So in the case of a sequential processor algorithm with 2 outputs and 3 input data, the evaluator write method will be call 6 times.
After talking with @andre.anjos, the current solution proposed is to add the output name written to as a parameter of the evaluator write method so that the developer can chose on which one he wants to "synchronize" the evaluator output.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/26Validation fails with utf-8 error2019-09-25T09:25:30ZSamuel GAISTValidation fails with utf-8 errorUnder some circumstances the values sent for validation may trigger an "utf-8" decoding error.
This comes from the fact that all zmq packets received are decoded before being passed to the callbacks corresponding to the received command...Under some circumstances the values sent for validation may trigger an "utf-8" decoding error.
This comes from the fact that all zmq packets received are decoded before being passed to the callbacks corresponding to the received command.
The solution here is to pass the received data as is to the callbacks and they are responsible for decoding the data if needed.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/25Handle duplicate key in json data2019-06-17T05:48:34ZSamuel GAISTHandle duplicate key in json dataCurrently loading a json file that contains a key multiple times will result in the last entry being used.
To avoid getting strange result, implement a hook that will raise an error and stop there.Currently loading a json file that contains a key multiple times will result in the last entry being used.
To avoid getting strange result, implement a hook that will raise an error and stop there.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/24Assert usage cleanup2019-05-10T08:16:51ZSamuel GAISTAssert usage cleanupFollowing bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.backend.python.
Relates to beat/beat.core#72Following bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.backend.python.
Relates to beat/beat.core#72Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/23Improve storage classes2019-05-08T08:53:02ZSamuel GAISTImprove storage classesThe Storage and CodeStorage classes share a parallel goals, part of their code is exactly the same yet they are two distinct classes.
Fix this by creating a common base class so they share the configuration member variable and their check.The Storage and CodeStorage classes share a parallel goals, part of their code is exactly the same yet they are two distinct classes.
Fix this by creating a common base class so they share the configuration member variable and their check.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/22Update schema tests to conform to beat.core!652019-04-25T07:08:43ZJaden DIEFENBAUGHUpdate schema tests to conform to beat.core!65The algorithm/plotter schemas have been refined in beat.core!65 to more closely reflect the actual restrictions of these object types. This means that some valid metadata tests are now invalid, and needs to be updated.
This is blocking ...The algorithm/plotter schemas have been refined in beat.core!65 to more closely reflect the actual restrictions of these object types. This means that some valid metadata tests are now invalid, and needs to be updated.
This is blocking beat.core!65 (see the [note about the failing test](https://gitlab.idiap.ch/beat/beat.core/merge_requests/65#note_41115) for info on how this was found)Jaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/21Add BEAT classifier to setup.py2019-04-11T08:58:17ZSamuel GAISTAdd BEAT classifier to setup.pyThe classifier has been added to Pypi so it now can be used.
See #19The classifier has been added to Pypi so it now can be used.
See #19https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/20Implement protocol templates2019-04-18T09:59:31ZSamuel GAISTImplement protocol templatesFollowing beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplicat...Following beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplication and therefore potential errors in the database definitions.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/19The PyPI trove classifier "Framework :: BEAT" does not work2019-04-11T07:11:40ZAndré AnjosThe PyPI trove classifier "Framework :: BEAT" does not workThe deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051The deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/18Remove macOS Python 2 build2018-09-28T10:06:05ZSamuel GAISTRemove macOS Python 2 buildSince the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
...Since the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
See https://gitlab.idiap.ch/bob/bob.admin/merge_requests/105Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/17Change license to BSD 3-clause2019-03-13T08:19:09ZAndré AnjosChange license to BSD 3-clauseAs discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core ...As discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core BEAT packages.
@samuel.gaist: could you please do that *before* the release? The idea is to go from AGPLv3 to BSD 3-clause (as the rest of Bob) for all packages up to beat.cmdline. We exclude beat.web from this, leaving it as AGPLv3.
We should do the following, in details:
1. Remove the file `LICENSE.AGPL`
2. Create a new file called `LICENSE` the contains the stock BSD 3-clause text (see info here: https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates)
3. Modify the file `MANIFEST.in` to include `LICENSE` instead of `LICENSE.AGPL`
4. Modify the file `setup.py` to list the license correctly on the classifier list (replace license key by `License :: OSI Approved :: BSD License`)
5. Modify the file `conda/meta.yaml` to set the license appropriately. Should read `license: BSD 3-Clause` and `license_family: BSD` on the `about` section, replacing the current AGPL stuff.
Please consult with @philip.abbet since I think rechecked, by then, the feasibility of this move and found no issues against doing this.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/16The input to a block gets the object from index method of View while the name...2018-09-18T12:27:44ZZohreh MOSTAANIThe input to a block gets the object from index method of View while the name from get method of ViewI am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (usin...I am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (using the grandtest protocol). I get the following error in both cases.
```
(beat_py3) zmostaani@helvetix02:/remote/idiap.svm/user.active/zmostaani/experiments/beat$ beat exp run zmostaani/zmostaani/sbhatta-iqm-face-antispoofing-test/1/replay2-antispoofing-iqm-lda-2
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Running `zmostaani/iqm_features/1' for block `Feats_IQM_Train'
Start the execution of 'zmostaani/iqm_features/1'
Traceback (most recent call last):
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/bin/beat", line 11, in <module>
sys.exit(main())
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/decorators.py", line 75, in _decorator
value = view_func(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 569, in run
return run_experiment(config, name, force, docker, local, quiet)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 212, in run_experiment
with executor:
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 182, in __enter__
self._prepare_inputs()
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 194, in _prepare_inputs
databases=self.databases
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/helpers.py", line 195, in create_inputs_from_configuration
start_index=start_index, end_index=end_index)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 174, in setup
start_index=start_index, end_index=end_index, pack=pack)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/data.py", line 569, in setup
previous_value = getattr(obj, output_name)
AttributeError: 'Entry' object has no attribute 'class'
```
In get method there is 'class' but in the index there is not, because beat is using namedtuple for making the Entry points and this function does not allow to have name 'class' as an Entry. Even if I define a dictionary instead of namedtuple in my database the python backend is chainging it to namedtuple.
```
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 157, in setup
Entry = namedtuple('Entry', sorted(objs[0].keys()))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/collections/__init__.py", line 404, in namedtuple
'keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'class'
```
I have two questions
- why not use simple dictionaries in python backend instead of namedtupe
- why the input of block is trying to access objects in index instead of get.
Please let me know if you need more information.