beat.backend.python issueshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues2017-08-08T04:40:37Zhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/1IO synchronization is managed twice2017-08-08T04:40:37ZLaurent EL SHAFEYIO synchronization is managed twiceThe I/O synchronization is currently manage twice: in `beat.io` and in `beat.backend.python`.
Since `beat.io` is common to any backend, it would be nice to only manage I/O synchronization in this package to keep the backends as simple...The I/O synchronization is currently manage twice: in `beat.io` and in `beat.backend.python`.
Since `beat.io` is common to any backend, it would be nice to only manage I/O synchronization in this package to keep the backends as simple as possible.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/2Move dataformat from beat.core2017-08-08T04:40:37ZLaurent EL SHAFEYMove dataformat from beat.coreIdeally, we don't want this package to depend on `beat.core`. But it requires dataformat.
One possible solution would be to move the dataformat from beat.core into this package, and to make beat.core depeonds on this package.Ideally, we don't want this package to depend on `beat.core`. But it requires dataformat.
One possible solution would be to move the dataformat from beat.core into this package, and to make beat.core depeonds on this package.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/3Properly define IO daemon address2017-08-08T04:40:37ZLaurent EL SHAFEYProperly define IO daemon addressThe address of the IO daemon is currently hard-coded as localhost.
This should be properly defined and passed as an argument of the execute script.The address of the IO daemon is currently hard-coded as localhost.
This should be properly defined and passed as an argument of the execute script.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/4Statistics split2017-08-08T04:40:37ZLaurent EL SHAFEYStatistics splitThe statistics are now generated by two different processes: the I/O daemon and the user process. However, everything is handled in one class, which means that statistics objects are often created but only half populated.
We may conside...The statistics are now generated by two different processes: the I/O daemon and the user process. However, everything is handled in one class, which means that statistics objects are often created but only half populated.
We may consider to split the class definition of statitics object to better reflect the underlying implementation.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/5Parameters handling requires a string to numpy-type conversion2017-08-08T04:40:37ZLaurent EL SHAFEYParameters handling requires a string to numpy-type conversionHandling the parameters of an algorithm requires the backend to be able to perform a string to numpy-type conversion.
This is currently performed by the method _convert_parameter_types() of the beat.core.algorithm.Algorithm class. Since...Handling the parameters of an algorithm requires the backend to be able to perform a string to numpy-type conversion.
This is currently performed by the method _convert_parameter_types() of the beat.core.algorithm.Algorithm class. Since the backend does not use this class, we need to move this function somewhere else to deal with experiment parameters.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/6Move algorithm and library into this package2017-08-08T04:40:37ZLaurent EL SHAFEYMove algorithm and library into this packageLaurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/7Do not open input/output pipe at each request2017-08-08T04:40:37ZLaurent EL SHAFEYDo not open input/output pipe at each requestAs discussed with @andre.anjos , this would be too inefficient.As discussed with @andre.anjos , this would be too inefficient.Laurent EL SHAFEYLaurent EL SHAFEYhttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/8Split beat.core and beat.backend.python2017-08-08T04:40:37ZLaurent EL SHAFEYSplit beat.core and beat.backend.pythonFollowing a discussion with @andre.anjos , few beat.core functionalities need to be in beat.backend.python.
To achieve this, we need to:
1. Move all the relevant code from beat.core to beat.backend.python
2. Make beat.core depends o...Following a discussion with @andre.anjos , few beat.core functionalities need to be in beat.backend.python.
To achieve this, we need to:
1. Move all the relevant code from beat.core to beat.backend.python
2. Make beat.core depends on beat.backend.python
3. Make all packages use features from beat.backend.python (when they were moved from beat.core to beat.backend.python)André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/9Remove prefix access and pass everything by value/json2017-08-08T04:40:37ZLaurent EL SHAFEYRemove prefix access and pass everything by value/jsonThe user process still has access to the prefix to fetch dataformats/algorithms/etc. Ideally, we need to get rid of this and to pass everything by value/json.The user process still has access to the prefix to fetch dataformats/algorithms/etc. Ideally, we need to get rid of this and to pass everything by value/json.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/10Improve test-ability of this package2017-08-08T04:40:37ZAndré AnjosImprove test-ability of this packageAs of today, it is not very easy to test the user process w/o an I/O daemon. We'd have to figure out a way to build unit tests around this package.As of today, it is not very easy to test the user process w/o an I/O daemon. We'd have to figure out a way to build unit tests around this package.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/11Create PipeSource/Sink instead of changing Output class2017-08-08T04:40:37ZAndré AnjosCreate PipeSource/Sink instead of changing Output classWe need to revert the design currently available on this package so that it respects the old beat.core design. Instead of overwriting the Input/Output classes, we need to create both a PipeSource and a PipeSink classes that can be used t...We need to revert the design currently available on this package so that it respects the old beat.core design. Instead of overwriting the Input/Output classes, we need to create both a PipeSource and a PipeSink classes that can be used to input and output data for the Input/Output classes respectively.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/12Parameter check on Algorithm class is not necessary2017-08-08T04:40:37ZAndré AnjosParameter check on Algorithm class is not necessaryAs soon as possible, remove this check as this is done by the BEAT infrastructure already.As soon as possible, remove this check as this is done by the BEAT infrastructure already.Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/13Memory leaks in the Python backend2017-08-08T04:40:36ZAndré AnjosMemory leaks in the Python backendAlthough most memory leaks have been mitigated with the release 0.1.3 of this package (optimization of `struct.pack` into `numpy.tostring`/`numpy.tobytes`), there seems still to exist a very mild memory leak. It is visible when one execu...Although most memory leaks have been mitigated with the release 0.1.3 of this package (optimization of `struct.pack` into `numpy.tostring`/`numpy.tobytes`), there seems still to exist a very mild memory leak. It is visible when one executes processes that access large amounts of data (such as in the Replay Attack dataset).
This problems needs to be investigated in a more closed setting for a leak-free execution.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/14Security improvements2017-10-16T09:47:16ZAndré AnjosSecurity improvementsThis bug tracks improvements to the security for running user algorithms inside the beat backend.
The possible best way to handle this would be with RestrictedPython (see this SO thread http://stackoverflow.com/questions/1350466/preve...This bug tracks improvements to the security for running user algorithms inside the beat backend.
The possible best way to handle this would be with RestrictedPython (see this SO thread http://stackoverflow.com/questions/1350466/preventing-python-code-from-importing-certain-modules).Philip ABBETPhilip ABBEThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/15no CI for master branch2018-09-18T12:32:20ZJaden DIEFENBAUGHno CI for master branchThere's no `gitlab-ci.yml` fileThere's no `gitlab-ci.yml` filehttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/16The input to a block gets the object from index method of View while the name...2018-09-18T12:27:44ZZohreh MOSTAANIThe input to a block gets the object from index method of View while the name from get method of ViewI am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (usin...I am running an experiment locally using beat.editor. I defined the replay-attack database (using both my own code and the one in beat.examples (https://gitlab.idiap.ch/beat/beat.examples/blob/master/advanced/databases/replay/4.py) (using the grandtest protocol). I get the following error in both cases.
```
(beat_py3) zmostaani@helvetix02:/remote/idiap.svm/user.active/zmostaani/experiments/beat$ beat exp run zmostaani/zmostaani/sbhatta-iqm-face-antispoofing-test/1/replay2-antispoofing-iqm-lda-2
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Index for database replay/2 not found, building it
Running `zmostaani/iqm_features/1' for block `Feats_IQM_Train'
Start the execution of 'zmostaani/iqm_features/1'
Traceback (most recent call last):
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/bin/beat", line 11, in <module>
sys.exit(main())
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/decorators.py", line 75, in _decorator
value = view_func(*args, **kwargs)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 569, in run
return run_experiment(config, name, force, docker, local, quiet)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/cmdline/experiments.py", line 212, in run_experiment
with executor:
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 182, in __enter__
self._prepare_inputs()
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/core/execution/local.py", line 194, in _prepare_inputs
databases=self.databases
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/helpers.py", line 195, in create_inputs_from_configuration
start_index=start_index, end_index=end_index)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 174, in setup
start_index=start_index, end_index=end_index, pack=pack)
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/data.py", line 569, in setup
previous_value = getattr(obj, output_name)
AttributeError: 'Entry' object has no attribute 'class'
```
In get method there is 'class' but in the index there is not, because beat is using namedtuple for making the Entry points and this function does not allow to have name 'class' as an Entry. Even if I define a dictionary instead of namedtuple in my database the python backend is chainging it to namedtuple.
```
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/site-packages/beat/backend/python/database.py", line 157, in setup
Entry = namedtuple('Entry', sorted(objs[0].keys()))
File "/idiap/user/zmostaani/miniconda3/envs/beat_py3/lib/python3.6/collections/__init__.py", line 404, in namedtuple
'keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'class'
```
I have two questions
- why not use simple dictionaries in python backend instead of namedtupe
- why the input of block is trying to access objects in index instead of get.
Please let me know if you need more information.https://gitlab.idiap.ch/beat/beat.backend.python/-/issues/17Change license to BSD 3-clause2019-03-13T08:19:09ZAndré AnjosChange license to BSD 3-clauseAs discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core ...As discussed in earlier weekly meetings (see action points from week02/2018 till week09/2018), the prospective re-use of BEAT core components to re-implement some Bob frameworks would require a shift in the licensing terms for some core BEAT packages.
@samuel.gaist: could you please do that *before* the release? The idea is to go from AGPLv3 to BSD 3-clause (as the rest of Bob) for all packages up to beat.cmdline. We exclude beat.web from this, leaving it as AGPLv3.
We should do the following, in details:
1. Remove the file `LICENSE.AGPL`
2. Create a new file called `LICENSE` the contains the stock BSD 3-clause text (see info here: https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates)
3. Modify the file `MANIFEST.in` to include `LICENSE` instead of `LICENSE.AGPL`
4. Modify the file `setup.py` to list the license correctly on the classifier list (replace license key by `License :: OSI Approved :: BSD License`)
5. Modify the file `conda/meta.yaml` to set the license appropriately. Should read `license: BSD 3-Clause` and `license_family: BSD` on the `about` section, replacing the current AGPL stuff.
Please consult with @philip.abbet since I think rechecked, by then, the feasibility of this move and found no issues against doing this.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/18Remove macOS Python 2 build2018-09-28T10:06:05ZSamuel GAISTRemove macOS Python 2 buildSince the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
...Since the BOB infrastructure is used to build the BEAT conda packages and it is planned to drop Python 2 support by the end of the year, the macOS builds are already planned to be removed shortly, therefore remove it from this package.
See https://gitlab.idiap.ch/bob/bob.admin/merge_requests/105Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/19The PyPI trove classifier "Framework :: BEAT" does not work2019-04-11T07:11:40ZAndré AnjosThe PyPI trove classifier "Framework :: BEAT" does not workThe deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051The deployment of beat packages in PyPI is broken because they all contain `Framework :: BEAT`. I think we should remove this globally so that the deployment works.
Example: https://gitlab.idiap.ch/beat/beat.backend.python/pipelines/24051Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.backend.python/-/issues/20Implement protocol templates2019-04-18T09:59:31ZSamuel GAISTImplement protocol templatesFollowing beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplicat...Following beat/beat.core#64 this issue tracks the implementation here for the support of the ProtocolTemplate object which will represent a database template.
The end goal is to make protocol template re-usable objects to avoid duplication and therefore potential errors in the database definitions.Samuel GAISTSamuel GAIST