beat.web issueshttps://gitlab.idiap.ch/beat/beat.web/-/issues2017-08-06T16:44:52Zhttps://gitlab.idiap.ch/beat/beat.web/-/issues/428[reports] chevrons tables2017-08-06T16:44:52ZFlavio TARSETTI[reports] chevrons tablesAdd gray up/down chevrons for all th in tables
Chevron need to be with white-space:nowrap next to thAdd gray up/down chevrons for all th in tables
Chevron need to be with white-space:nowrap next to thOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/427[reports] patch for sorting items + total time sort + experiment names2017-08-06T16:44:52ZFlavio TARSETTI[reports] patch for sorting items + total time sort + experiment namesPatch required for sorting itemsPatch required for sorting itemsOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/426[reports] tables sorting and saving2017-08-06T16:44:52ZFlavio TARSETTI[reports] tables sorting and savingThe tables in reports are currently not sortable.
We would like to enforce a mechanism to sort and save this sorting preferences for each tables of the report.
Others viewers should also be able to sort the table if they wish too.The tables in reports are currently not sortable.
We would like to enforce a mechanism to sort and save this sorting preferences for each tables of the report.
Others viewers should also be able to sort the table if they wish too.Open-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/425[algorithms] If algorithms are only usable, users can still see it through th...2017-08-06T16:44:52ZAndré Anjos[algorithms] If algorithms are only usable, users can still see it through the "diff" panelExample:
https://www.beat-eu.org/platform/algorithms/diff/anjithgeorge/KBOC_IITKGP_ANJITH_APRIL7_1/10/anjithgeorge/KBOC_IITKGP_ANJITH_APRIL7_1/14/
This option should be removed.Example:
https://www.beat-eu.org/platform/algorithms/diff/anjithgeorge/KBOC_IITKGP_ANJITH_APRIL7_1/10/anjithgeorge/KBOC_IITKGP_ANJITH_APRIL7_1/14/
This option should be removed.Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.web/-/issues/423[report] reload on saved and unified saving action with the rest of the report2017-08-06T16:44:52ZFlavio TARSETTI[report] reload on saved and unified saving action with the rest of the reportTo make sure everything is saved properly, it's better to reload the page after a save.To make sure everything is saved properly, it's better to reload the page after a save.Open-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/422[reports] on multiple aliasing change avoid error with database2017-08-06T16:44:52ZFlavio TARSETTI[reports] on multiple aliasing change avoid error with databaseOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/421[reports] fix alias issue2017-08-06T16:44:52ZFlavio TARSETTI[reports] fix alias issueOn dynamic aliasing, the experiment was not found and set to plotter as the alias changesOn dynamic aliasing, the experiment was not found and set to plotter as the alias changesOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/435[experiments] Arrow icons not in the right direction2017-08-06T16:44:51ZAndré Anjos[experiments] Arrow icons not in the right directionThere is something wrong with the current logic:
![image](/uploads/6917f0a8a9e6382a3964868089b05326/image.png)
There is something wrong with the current logic:
![image](/uploads/6917f0a8a9e6382a3964868089b05326/image.png)
Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.web/-/issues/406[teams] Teams can be deleted, even if there are resources which are shared th...2016-09-16T09:35:04ZAndré Anjos[teams] Teams can be deleted, even if there are resources which are shared through themThere is, currently, some inconsistence with the team deletion property and sharing.
If a contribution is shared via a team with someone, then the contribution is marked as "Shared" (sharing status) and the team is add to its ``shared...There is, currently, some inconsistence with the team deletion property and sharing.
If a contribution is shared via a team with someone, then the contribution is marked as "Shared" (sharing status) and the team is add to its ``shared_with_team`` list.
The owner of the team may now proceed and delete the team, which will leave the contribution marked with ``Shared``, but now with no one on the ``shared_with_team`` list.
If the user has used the contribution, I'm not sure in which status that leaves us.
My understanding for the simplest way out is this: if a team has been used to share a contribution, it can no longer be deleted.Open-source Releasehttps://gitlab.idiap.ch/beat/beat.web/-/issues/7Integration with the new scheduler2016-06-24T10:33:06ZAndré AnjosIntegration with the new schedulerHere is a task list with topics that require attention for the integration:
- [X] The most important feature to be integrated is the ability to start/stop the experiment. This has been implemented in the scheduler.
- [X] When deletin...Here is a task list with topics that require attention for the integration:
- [X] The most important feature to be integrated is the ability to start/stop the experiment. This has been implemented in the scheduler.
- [X] When deleting an experiment, if the experiment is still running, then the experiment must be cancelled by the scheduler
- [X] The update of the dataflow drawing while the toolchain executes should respect the new abilities of the scheduler to report start/end of each block and for the experiment as they arrive. There is no need anymore for the platform to guess that certain blocks are being executed.
- [ ] There should be an administrative page on the beat web site that allows for:
- [X] Viewing the current scheduler state through the `/state` API call
- [X] Cancel all running experiments
- [X] Clean the cache content
- [ ] Change the scheduling policy
- [ ] Change the debugging level
- [X] Change the queue/environment/worker/slot configuration.
- [X] The web server should hold and allow for administrative change of queue/environment/worker/slot properties
- [x] For each block of the experiment, the user should be allowed to select:
- [x] Processing queue to use, following user rights
- [x] Processing environment to use
- [x] Number of slots to occupy
These information should be transmitted to the scheduler.
Hints on behavior and implementation details:
### Use Cases
1. Simple xp submission by an user with no particular rights
2. A xp submission with a user which has 1000 points of reputation
3. A xp submission with a user that has priority on a given queue
### New Tables on the Django DB:
We must first define an "environment". The environment has a name (e.g. "Python"), a version (e.g. "2.7.3"), a OS (e.g. "Debian Wheezy 7.2 (x86_64)"), a rich description string which defines all properties of that environment, including installed packages.
Each "queue" in the system is defined with a name (e.g. "3 hours/4G on Python"), a memory limit (e.g. "4096Mb"), a time limit (e.g. "3 hours"), an environment (e.g. "Python") and the number of slots the queue can occupy, at most, on every machine available in the system.)
Each user "library" consists of a bunch of files that are packaged together and put on a certain directory. Their organisation follows the same strategies as for algorithms. The sole exception is that each library is represented by a directory rather than a single file. The "library" is defined with a name (e.g. "lbp"), a version (e.g. "1.0"), an environment compatibility list and a list of other libraries this library depends on.
We must also define another table called "GroupQueueRights" in which we are going to track 4 aspects: queue - group - max slots - priority. For example: a user belonging to group "default" may be able to use 5 slots on queue "3 hours/4G on Python" with priority 0.
A user may belong to several groups. In this case, the platform should only consider the maximum slots/priority for each Queue when submitting the job to the scheduler.
E.g.: on GroupQueueRights table (queue name, group name, slots, priority):
```text
Row 1: Q1 - default - 2 - 0
Row 2: Q1 - special - 1 - 1
Row 3: Q1 - super - 3 - 1
User "A" belongs to group "default": Computed user queue rights are (Q1 - 2 - 0)
User "B" belongs to groups "default" and "super": Computed user queue rights are (Q1 - 3 - 1)
User "C" belongs to groups "default" and "special": Computed user queue rights are (Q1 - 2 - 1)
```
### Worker Perspective
The worker installed in each machine knows where each local environment with a given name and version is installed and how to execute user programs using that environment.
The program execution receives as parameters:
1. the environment
2. the parameters to call the environment executable with
> N.B.: The parameter list API for all environment executables defines our so-called Sandboxing
> API. It has to be respected for all environment implementations.
The worker is just told what to do - it does not check for rights or know any of that.
### User Perspective Operation
The user selects an overall execution queue for that experiment. It may also specify individual queues for individual blocks. The platform only allows the user to select queues in which the user has at least 1 slot for processing (other queues, even if they exist, are suppressed from the selection box).
After basic queue selection, every block on the toolchain executes in a single slot for the selected queues. If the algorithm does support a multi-slot operation, then the platform will allow the user to select how many slots (max'ed at the user rights) to use for a given block.
Users can submit as many jobs as they want. They will be treated according to queue rights and farm availability.
### Web Platform
In possession of all this information, the web platform will submit to the scheduler, the experiment for execution. Each request for execution contains 4 components:
* Toolchain
* Configuration, containing the queues, libraries and slots the user wants to deploy
* Username or ID
* User queue rights - computed from the max of all the groups the user belongs to
> NB: The reputation can be implemented as a multiplying factor. (ceilled at the total # of slots for a
> particular queue). For example, a user with Reputation = 200 has 2x more processing power than the established
> User queue rights.
### Scheduler
The scheduler will receive run-xp requests and breaks down the experiments in jobs (representing the blocks) with dependencies.
At each queue loop it must decide what to execute based on:
* Current slot availability for the different queues/users (user-queue occupation state needs to be stored)
* Job queueing time (how old is the job?)
* User queue rights/priority
> N.B.: If the Scheduler receives a second job for the same user, the queue rights for that user will be updated
> with the new values, in case they differ.
### Simplifications for first implementation
1. There is only 1 environment installed, based on Python/"execute_single_algorithm.py".
2. All queue priorities are set to 0 (i.e. the scheduler can ignore it)
3. #slots/memory goes in pairs until we understand all this a bit better
4. The scheduler implements only a FIFO (first queued/runnable is run) strategy based on the job submission age
5. Libraries must be written in pure Python
6. No reputation system is in place just yetOpen-source Releasehttps://gitlab.idiap.ch/beat/beat.web/-/issues/367[backend] Wipe cache and cancel experiment actions are not working2016-06-24T10:27:57ZAndré Anjos[backend] Wipe cache and cancel experiment actions are not workingCurrently, both result in a time out. Needs to test if the backend is working properly.Currently, both result in a time out. Needs to test if the backend is working properly.Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.web/-/issues/420[reports] merge/expand correct flag to be set2016-04-06T09:09:01ZFlavio TARSETTI[reports] merge/expand correct flag to be setOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/419[reports] unable to do a PUT on reports2016-04-06T09:09:00ZFlavio TARSETTI[reports] unable to do a PUT on reportsThis is due to the merge/expand that corrupts the selected_template in the database.This is due to the merge/expand that corrupts the selected_template in the database.Open-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/418[reports] plot_404 error2016-03-21T15:27:04ZFlavio TARSETTI[reports] plot_404 erroron editable reports, plotter seems to sometimes get no experiment informationon editable reports, plotter seems to sometimes get no experiment informationOpen-source ReleaseFlavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.web/-/issues/341[ui] Replace on the frontpage the example toolchain by the public activity feed2016-03-16T12:01:27ZSébastien MARCEL[ui] Replace on the frontpage the example toolchain by the public activity feedhttp://beatweb-staginghttp://beatweb-stagingOpen-source Releasehttps://gitlab.idiap.ch/beat/beat.web/-/issues/169Video tutorials2016-03-16T12:00:17ZAndré AnjosVideo tutorialsFor each set of key pages on the platform, it would be nice to have video tutorials explaining what those objects are for and how to use the site. This would complement well any tutorial material we may have.
Here is a preliminary lis...For each set of key pages on the platform, it would be nice to have video tutorials explaining what those objects are for and how to use the site. This would complement well any tutorial material we may have.
Here is a preliminary list of (short) videos that could be done:
- [ ] platform overview and organisation
- [ ] experiments (intro, list)
- [ ] experiments (configurator, launching)
- [ ] searching (and reporting when available)
- [ ] toolchains (intro, list, creation, edition)
- [ ] data formats (intro, list, creation, edition)
- [ ] algorithms (list, creation, edition)
- [ ] algorithms (synchronisation basics)
- [ ] libraries
- [ ] teams
- [ ] searching (how to organise a competition)
- [ ] searching (how to organise a study group)
- [ ] backend (environments, parallel processing)
The idea is that those videos are somehow easily accessible (linked) from the beat.web website in key locations.Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.web/-/issues/305Beat Licence2016-03-14T13:25:57ZHugues SALAMINBeat LicenceDear all,
as you may know, one of the Milestone in February 2016 for the Beat project is to release an open source version of the platform. The goal of this issue is to discuss:
1) What license will be used ?
2) How to handle co...Dear all,
as you may know, one of the Milestone in February 2016 for the Beat project is to release an open source version of the platform. The goal of this issue is to discuss:
1) What license will be used ?
2) How to handle contribution from people outside of Idiap ?
3) How to separate the modules that will be in the open-sourced version and the modules that will stay closed source ?
1) What license will be used ?
Idiap default license is the GPLv3. I think that in this case, using the GNU AGPLv3, makes more sense. By using the AGPL, anyone providing Beat as a service will also have to share his/her modifications. It is important to note that, as Idiap owns the copyright on all the code, the license we choose will not limit what Idiap can do with the platform. There will still be the possibility for Idiap to propose commercial license or produce derived produce not under the GPL/AGPL.
2) How to handle contribution from people outside of Idiap ?
I suggest doing a Copyright transfer agreement for external contributors. The Idea is the ensure that Idiap continue to own the copyright on the whole platform. Any other suggestions are welcome.
3) How to separate the modules that will be in the open-sourced version and the modules that are closed source ?
I have no suggestion here. As all of you know the architecture better than me, you are probably more competent to suggest a solution here.
Once we settle on a license, we will know what information need to be added to the files and directories in the project.
Cheers,
HuguesOpen-source Releasehttps://gitlab.idiap.ch/beat/beat.web/-/issues/414Experiments API issue ?2016-03-14T09:44:31ZFlavio TARSETTIExperiments API issue ?Some experiments are accessible, some not ?
This is strange:
Accessible:
http://beatweb-staging/api/v1/experiments/tutorial/tutorial/eigenface_with_preprocessing/1/atnt-eigenfaces-3-comp-preproc-para/
Not accessible:
http:...Some experiments are accessible, some not ?
This is strange:
Accessible:
http://beatweb-staging/api/v1/experiments/tutorial/tutorial/eigenface_with_preprocessing/1/atnt-eigenfaces-3-comp-preproc-para/
Not accessible:
http://beatweb-staging/api/v1/experiments/tutorial/tutorial/eigenface_with_preprocessing/1/atnt-eigenfaces-10-comp-tantriggs/Open-source ReleaseAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.web/-/issues/405[databases] Hitting API issues a 500 error2016-02-24T09:44:42ZAndré Anjos[databases] Hitting API issues a 500 errorFor some reason, hitting the databases API this way:
https://www.beat-eu.org/platform/api/v1/databases/?fields=name,hash,description,creation_date
Raises an internal error:
```text
Internal Server Error: /platform/api/v1/databa...For some reason, hitting the databases API this way:
https://www.beat-eu.org/platform/api/v1/databases/?fields=name,hash,description,creation_date
Raises an internal error:
```text
Internal Server Error: /platform/api/v1/databases/
Traceback (most recent call last):
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/views.py", line 466, in dispatch
response = self.handle_exception(exc)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
response = handler(request, *args, **kwargs)
File "/remote/sw/beat.web/src/beat.web/beat/web/databases/api.py", line 95, in get
return Response(serializer.data)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/serializers.py", line 674, in data
ret = super(ListSerializer, self).data
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/serializers.py", line 239, in data
self._data = self.to_representation(self.instance)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/serializers.py", line 614, in to_representation
self.child.to_representation(item) for item in iterable
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/serializers.py", line 472, in to_representation
ret[field.field_name] = field.to_representation(attribute)
File "/remote/sw/beat.env.web-1.0.0/usr/lib/python2.7/site-packages/rest_framework/fields.py", line 703, in to_representation
return six.text_type(value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 902: ordinal not in range(128)
Request repr():
<WSGIRequest
path:/platform/api/v1/databases/,
GET:<QueryDict: {u'fields': [u'name,hash,description,creation_date']}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '',
'CONTENT_TYPE': '',
'HTTP_ACCEPT_ENCODING': 'identity',
'HTTP_AUTHORIZATION': 'Token 1137793fe71df5d3451292e02a5bc288e6646e5c',
'HTTP_CONNECTION': 'Keep-Alive',
'HTTP_HOST': 'beatweb.lab.idiap.ch',
'HTTP_X_FORWARDED_FOR': '172.29.17.14',
'HTTP_X_FORWARDED_HOST': 'www.beat-eu.org',
'HTTP_X_FORWARDED_SERVER': 'www.beat-eu.org',
'PATH_INFO': u'/platform/api/v1/databases/',
'QUERY_STRING': 'fields=name,hash,description,creation_date',
'REMOTE_ADDR': '172.30.90.51',
'REQUEST_METHOD': 'GET',
u'SCRIPT_NAME': u'',
'SERVER_ADDR': '172.31.100.250',
'SERVER_NAME': 'www.beat-eu.org',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.1',
'X_FORWARDED_HOST': 'www.beat-eu.org',
'X_FORWARDED_PROTO': 'https',
'uwsgi.node': 'beatweb.lab.kvm',
'uwsgi.version': '2.0.12',
'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7f769d0f2930>,
'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
'wsgi.input': <uwsgi._Input object at 0x7f76993f6138>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)}>
```Open-source Releasehttps://gitlab.idiap.ch/beat/beat.web/-/issues/402[experiments] Print protocol names with a '.' instead of '@'2016-02-19T11:27:04ZAndré Anjos[experiments] Print protocol names with a '.' instead of '@'In the search filters, we must use a '.' instead of '@' for subselecting the protocol and this is confusing for the user, which may just cut-n-paste the value from the shown table and expect it to work.
Another option would be to fix...In the search filters, we must use a '.' instead of '@' for subselecting the protocol and this is confusing for the user, which may just cut-n-paste the value from the shown table and expect it to work.
Another option would be to fix the search so it does work with an '@' sign for the protocol separation.Open-source ReleaseAndré AnjosAndré Anjos