beat.backend.python issueshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues2020-06-08T14:00:57Zhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/32Multiprocessing support for data sources2020-06-08T14:00:57ZAndré AnjosMultiprocessing support for data sourcesAs discussed in today's debugging session with @samuel.gaist and @amohammadi, using a `DataLoader` object in a multiprocessing context is hard:
1. Typically, the underlying `DataSource`'s `fileobj`'s are opened by the time the process i...As discussed in today's debugging session with @samuel.gaist and @amohammadi, using a `DataLoader` object in a multiprocessing context is hard:
1. Typically, the underlying `DataSource`'s `fileobj`'s are opened by the time the process is forked
2. Deep copying the object (which goes through pickling and unpickling it) does not properly reset underlying `fileobj` pointers, which makes multiple processes access the same underlying OS-level file handler, causing unwanted behaviour.
To sort this out, we discussed 2 possible additions to this package:
1. `DataLoader` should have a `reset()` method that resets all underlying `DataSource` opened files, so that they can be correctly copied across multiple processes (e.g. in the event of a `fork()`). It should be relatively easy to do a `reset()` operation across all inputs of a user algorithm, to ensure all data sources are properly reset before an eventual user-guided `fork()`.
2. The underlying `DataSource` should have its pickle/unpickle behaviour patched (via overwriting the `__setstate__` slot of `DataSource`, see reference below), so that unpickling a data source (e.g. indirectly via a data loader deep copy), will call `self.reset()` after its state is unpickled. This would allow a `DataLoader` object to be sent over current mechanisms for inter-process communication (e.g. MPI or `multiprocessing.Queue`), transparently.
References:
* Python fileobj handling: https://stackoverflow.com/questions/1834556/does-a-file-object-automatically-close-when-its-reference-count-hits-zero
* Pickle user guide (see in particular `__getstate__` and `__setstate__` on how to overwrite the pickle/unpickle actions): https://docs.python.org/3/library/pickle.html#object.__getstate__
* On sharing (opened) file pointers in a POSIX system after a `fork()` is issued: https://stackoverflow.com/questions/33899548/file-pointers-after-returning-from-a-forked-child-processSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/30Error when waiting on a loop that wasn't started2020-02-06T17:36:09ZSamuel GAISTError when waiting on a loop that wasn't startedIf for some reason the loop, or database, `process` method was not called and `wait` is called, a runtime error will occur because the message handler was not started as expected by the current code.If for some reason the loop, or database, `process` method was not called and `wait` is called, a runtime error will occur because the message handler was not started as expected by the current code.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/28License file location is wrong in conda recipe2019-11-14T15:58:26ZAndré AnjosLicense file location is wrong in conda recipeIt should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.It should read `LICENSE` instead of `../LICENSE`.
The same should be propagated to all relevant packages.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/27Improve loop evaluator synchronized write2019-10-01T10:08:15ZSamuel GAISTImprove loop evaluator synchronized writeThe loop evaluator write method is call once per output write. However, a processor block may have several outputs which means that the evaluator write method will be called as many times as an output is written. So in the case of a sequ...The loop evaluator write method is call once per output write. However, a processor block may have several outputs which means that the evaluator write method will be called as many times as an output is written. So in the case of a sequential processor algorithm with 2 outputs and 3 input data, the evaluator write method will be call 6 times.
After talking with @andre.anjos, the current solution proposed is to add the output name written to as a parameter of the evaluator write method so that the developer can chose on which one he wants to "synchronize" the evaluator output.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/26Validation fails with utf-8 error2019-09-25T09:25:30ZSamuel GAISTValidation fails with utf-8 errorUnder some circumstances the values sent for validation may trigger an "utf-8" decoding error.
This comes from the fact that all zmq packets received are decoded before being passed to the callbacks corresponding to the received command...Under some circumstances the values sent for validation may trigger an "utf-8" decoding error.
This comes from the fact that all zmq packets received are decoded before being passed to the callbacks corresponding to the received command.
The solution here is to pass the received data as is to the callbacks and they are responsible for decoding the data if needed.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/25Handle duplicate key in json data2019-06-17T05:48:34ZSamuel GAISTHandle duplicate key in json dataCurrently loading a json file that contains a key multiple times will result in the last entry being used.
To avoid getting strange result, implement a hook that will raise an error and stop there.Currently loading a json file that contains a key multiple times will result in the last entry being used.
To avoid getting strange result, implement a hook that will raise an error and stop there.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/24Assert usage cleanup2019-05-10T08:16:51ZSamuel GAISTAssert usage cleanupFollowing bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.backend.python.
Relates to beat/beat.core#72Following bandit warning about usage of assert in code, this issue is used to track down the cleanup of these statements found in beat.backend.python.
Relates to beat/beat.core#72Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/23Improve storage classes2019-05-08T08:53:02ZSamuel GAISTImprove storage classesThe Storage and CodeStorage classes share a parallel goals, part of their code is exactly the same yet they are two distinct classes.
Fix this by creating a common base class so they share the configuration member variable and their check.The Storage and CodeStorage classes share a parallel goals, part of their code is exactly the same yet they are two distinct classes.
Fix this by creating a common base class so they share the configuration member variable and their check.Soft loopsSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/22Update schema tests to conform to beat.core!652019-04-25T07:08:43ZJaden DIEFENBAUGHUpdate schema tests to conform to beat.core!65The algorithm/plotter schemas have been refined in beat.core!65 to more closely reflect the actual restrictions of these object types. This means that some valid metadata tests are now invalid, and needs to be updated.
This is blocking ...The algorithm/plotter schemas have been refined in beat.core!65 to more closely reflect the actual restrictions of these object types. This means that some valid metadata tests are now invalid, and needs to be updated.
This is blocking beat.core!65 (see the [note about the failing test](https://gitlab.idiap.ch/beat/beat.core/merge_requests/65#note_41115) for info on how this was found)Jaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/20Implement protocol templates2019-04-18T09:59:31ZSamuel GAISTImplement protocol templatesFollowing beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplicat...Following beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplication and therefore potential errors in the database definitions.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/21Add BEAT classifier to setup.py2019-04-11T08:58:17ZSamuel GAISTAdd BEAT classifier to setup.pyThe classifier has been added to Pypi so it now can be used.
See #19The classifier has been added to Pypi so it now can be used.
See #19https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/19The PyPI trove classifier "Framework :: BEAT" does not work2019-04-11T07:11:40ZAndré AnjosThe PyPI trove classifier "Framework :: BEAT" does not workThe deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051The deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/17Change license to BSD 3-clause2019-03-13T08:19:09ZAndré AnjosChange license to BSD 3-clauseAs discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core ...As discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core BEAT packages.
@samuel.gaist: could you please do that *before* the release? The idea is to go from AGPLv3 to BSD 3-clause (as the rest of Bob) for all packages up to beat.cmdline. We exclude beat.web from this, leaving it as AGPLv3.
We should do the following, in details:
1. Remove the file `LICENSE.AGPL`
2. Create a new file called `LICENSE` the contains the stock BSD 3-clause text (see info here: https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates)
3. Modify the file `MANIFEST.in` to include `LICENSE` instead of `LICENSE.AGPL`
4. Modify the file `setup.py` to list the license correctly on the classifier list (replace license key by `License :: OSI Approved :: BSD License`)
5. Modify the file `conda/meta.yaml` to set the license appropriately. Should read `license: BSD 3-Clause` and `license_family: BSD` on the `about` section, replacing the current AGPL stuff.
Please consult with @philip.abbet since I think rechecked, by then, the feasibility of this move and found no issues against doing this.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/18Remove macOS Python 2 build2018-09-28T10:06:05ZSamuel GAISTRemove macOS Python 2 buildSince the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
...Since the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
See https://gitlab.idiap.ch/bob/bob.admin/merge_requests/105Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/15no CI for master branch2018-09-18T12:32:20ZJaden DIEFENBAUGHno CI for master branchThere's no `gitlab-ci.yml` fileThere's no `gitlab-ci.yml` filehttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/16The input to a block gets the object from index method of View while the name...2018-09-18T12:27:44ZZohreh MOSTAANIThe input to a block gets the object from index method of View while the name from get method of ViewI am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (usin...I am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (using the grandtest protocol). I get the following error in both cases.
```
(beat_py3) zmostaani@helvetix02:/remote/idiap.svm/user.active/zmostaani/experiments/beat$ beat exp run zmostaani/zmostaani/sbhatta-iqm-face-antispoofing-test/1/replay2-antispoofing-iqm-lda-2
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Running `zmostaani/iqm_features/1' for block `Feats_IQM_Train'
Start the execution of 'zmostaani/iqm_features/1'
Traceback (most recent call last):
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/bin/beat", line 11, in <module>
sys.exit(main())
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/decorators.py", line 75, in _decorator
value = view_func(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 569, in run
return run_experiment(config, name, force, docker, local, quiet)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 212, in run_experiment
with executor:
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 182, in __enter__
self._prepare_inputs()
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 194, in _prepare_inputs
databases=self.databases
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/helpers.py", line 195, in create_inputs_from_configuration
start_index=start_index, end_index=end_index)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 174, in setup
start_index=start_index, end_index=end_index, pack=pack)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/data.py", line 569, in setup
previous_value = getattr(obj, output_name)
AttributeError: 'Entry' object has no attribute 'class'
```
In get method there is 'class' but in the index there is not, because beat is using namedtuple for making the Entry points and this function does not allow to have name 'class' as an Entry. Even if I define a dictionary instead of namedtuple in my database the python backend is chainging it to namedtuple.
```
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 157, in setup
Entry = namedtuple('Entry', sorted(objs[0].keys()))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/collections/__init__.py", line 404, in namedtuple
'keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'class'
```
I have two questions
- why not use simple dictionaries in python backend instead of namedtupe
- why the input of block is trying to access objects in index instead of get.
Please let me know if you need more information.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/14Security improvements2017-10-16T09:47:16ZAndré AnjosSecurity improvementsThis bug tracks improvements to the security for running user algorithms inside the beat backend.
The possible best way to handle this would be with RestrictedPython (see this SO thread http://stackoverflow.com/questions/1350466/preve...This bug tracks improvements to the security for running user algorithms inside the beat backend.
The possible best way to handle this would be with RestrictedPython (see this SO thread http://stackoverflow.com/questions/1350466/preventing-python-code-from-importing-certain-modules).Philip ABBETPhilip ABBEThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/12Parameter check on Algorithm class is not necessary2017-08-08T04:40:37ZAndré AnjosParameter check on Algorithm class is not necessaryAs soon as possible, remove this check as this is done by the BEAT infrastructure already.As soon as possible, remove this check as this is done by the BEAT infrastructure already.Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/11Create PipeSource/Sink instead of changing Output class2017-08-08T04:40:37ZAndré AnjosCreate PipeSource/Sink instead of changing Output classWe need to revert the design currently available on this package so that it respects the old beat.core design. Instead of overwriting the Input/Output classes, we need to create both a PipeSource and a PipeSink classes that can be used t...We need to revert the design currently available on this package so that it respects the old beat.core design. Instead of overwriting the Input/Output classes, we need to create both a PipeSource and a PipeSink classes that can be used to input and output data for the Input/Output classes respectively.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/10Improve test-ability of this package2017-08-08T04:40:37ZAndré AnjosImprove test-ability of this packageAs of today, it is not very easy to test the user process w/o an I/O daemon. We'd have to figure out a way to build unit tests around this package.As of today, it is not very easy to test the user process w/o an I/O daemon. We'd have to figure out a way to build unit tests around this package.André AnjosAndré Anjos