beat issueshttps://gitlab.idiap.ch/groups/beat/-/issues2020-02-12T13:04:47Zhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/93Show experiment execution status2020-02-12T13:04:47ZJaden DIEFENBAUGHShow experiment execution statusSorta like `beat.web` - let the user watch/review execution status~info on the webappSorta like `beat.web` - let the user watch/review execution status~info on the webapphttps://gitlab.idiap.ch/beat/beat.web/-/issues/497Move to conda package2018-10-30T15:28:49ZSamuel GAISTMove to conda packageCurrently all beat.something dependencies are being refactored to become conda packages. It has already been done for [beat.backend.python](https://gitlab.idiap.ch/beat/beat.backend.python/), [beat.core](https://gitlab.idiap.ch/beat/beat...Currently all beat.something dependencies are being refactored to become conda packages. It has already been done for [beat.backend.python](https://gitlab.idiap.ch/beat/beat.backend.python/), [beat.core](https://gitlab.idiap.ch/beat/beat.core/) and [beat.cmdline](https://gitlab.idiap.ch/beat/beat.cmdline/). The only one missing is [beat.examples](https://gitlab.idiap.ch/beat/beat.examples/).
beat.web poses a different challenge has it requires several dependencies that are not directly available through conda's default channel or Idiaps software channel for bob. The main challenge comes from the fact that `conda build` doesn't support dependencies from PyPI directly. The current recommend practice is to use `conda skeleton` to create recipes for PyPI packages and use them to feed `conda build`
Here is a list of the dependencies that can be found in several places in `beat.web` like `setup.py` and `buildout.cfg` with their requirements and conda channel they can be found in if any:
| package name | requirements | channel | remarques |
|--------------|--------------|---------|-----------|
|beat.core||idiap|
|alabaster||pkgs/main,pkgs/free|
|babel||pkgs/main,pkgs/free|
|backports.shutil-get-terminal-size||ODSP-TEST|
|colorlog||software/bob/conda|
|coverage||pkgs/main,pkgs/free|
|cycler||pkgs/main,pkgs/free|
|decorator||pkgs/main,pkgs/free|
|django|>=1.11,<2.0|defaults|
|django-activity-stream|>=0.6.5||too old|
|django-rest-swagger|>2.1||too old|
|django-guardian|>=1.3|conda-forge|
|djangorestframework|>3.7|conda-forge|
|django-jsonfield|>=1.0.1||too old|
|django-jsonfield-compat||None|
|docopt||pkgs/free|
|docutils||pkgs/free|
|funcsigs||pkgs/main,pkgs/free|
|functools32||pkgs/main,pkgs/free|
|graphviz||pkgs/main,pkgs/free|
|ipdb||software/bob/conda|
|ipython||pkgs/main,pkgs/free|
|ipython-genutils||GlaxoSmithKline|linux only|
|imagesize||pkgs/main,pkgs/free|
|Jinja2||pkgs/main,pkgs/free|
|jsonschema||pkgs/main,pkgs/free|
|MarkupSafe||pkgs/main,pkgs/free|
|matplotlib||pkgs/main,pkgs/free|
|mock||pkgs/main,pkgs/free|
|nose||pkgs/main,pkgs/free|
|numpy||pkgs/main,pkgs/free|
|oset||software/bob/conda|
|pathlib2||pkgs/main,pkgs/free|
|pbr||pkgs/main,pkgs/free|
|pexpect||pkgs/main,pkgs/free|
|pickleshare||pkgs/main,pkgs/free|
|pillow||pkgs/main,pkgs/free|
|pip||pkgs/main,pkgs/free|
|psutil||pkgs/main,pkgs/free|
|ptyprocess||pkgs/main,pkgs/free|
|psycopg2||pkgs/main,pkgs/free|
|Pygments||pkgs/main,pkgs/free|
|pyparsing||pkgs/main,pkgs/free|
|python-archive||auto|linux-only|
|python-dateutil||pkgs/main,pkgs/free|
|pytz||pkgs/main,pkgs/free|
|PyYAML||pkgs/main,pkgs/free|
|pyzmq||pkgs/main,pkgs/free|
|simplegeneric||pkgs/main,pkgs/free|
|simplejson||pkgs/main,pkgs/free|
|six||pkgs/main,pkgs/free|
|snowballstemmer||pkgs/main,pkgs/free|
|sphinx||pkgs/main,pkgs/free|
|sphinxcontrib-ansi|||None|
|sphinxcontrib-httpdomain||agoodman|
|sphinxcontrib-mscgen|||None|
|sphinxcontrib-programoutput||software/bob/conda|
|sphinx-rtd-theme|||None|
|termcolor||pkgs/main,pkgs/free|
|traitlets||pkgs/main,pkgs/free|
|uwsgi||conda-forge|
One question that can be asked here is: do we still need all of them ?
This will also fix #462 Gitlab CI IntegrationSamuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.editor/-/issues/92Remove prefix setting in webapp2018-05-02T15:28:28ZJaden DIEFENBAUGHRemove prefix setting in webappSince `beat.cmdline` is now a dependency there's no reason to have thisSince `beat.cmdline` is now a dependency there's no reason to have thishttps://gitlab.idiap.ch/beat/beat.editor/-/issues/91Auto-launch new browser tab when starting `beateditor` script2018-05-04T13:11:00ZJaden DIEFENBAUGHAuto-launch new browser tab when starting `beateditor` scriptAndré AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.editor/-/issues/90Separate cloning & versioning functionality in list views2018-05-18T14:29:39ZJaden DIEFENBAUGHSeparate cloning & versioning functionality in list viewsRight now there's only a "clone" button that does double duty.
There should be a "clone" button that requires a new object name and pins the version to "1"
There should be a "new version" button that pins the name and ups the version b...Right now there's only a "clone" button that does double duty.
There should be a "clone" button that requires a new object name and pins the version to "1"
There should be a "new version" button that pins the name and ups the version by onehttps://gitlab.idiap.ch/beat/beat.editor/-/issues/89Collapse objects' versions in lists2020-02-12T12:15:12ZJaden DIEFENBAUGHCollapse objects' versions in listsInstead of showing all the versions of objects by default, collapse them together and only show the newest version
would need ux for seeing the other versionsInstead of showing all the versions of objects by default, collapse them together and only show the newest version
would need ux for seeing the other versionshttps://gitlab.idiap.ch/beat/beat.editor/-/issues/88Remove "Documentation" tab, add "Info" & "Code" tabs2019-02-12T18:51:14ZJaden DIEFENBAUGHRemove "Documentation" tab, add "Info" & "Code" tabsInfo tab will have the RST file contents & its file path
Code tab will have the python file contents & its file path
Might eventually add RST rendering and/or in-browser editors
Would also need a "refresh" button on both to refresh fi...Info tab will have the RST file contents & its file path
Code tab will have the python file contents & its file path
Might eventually add RST rendering and/or in-browser editors
Would also need a "refresh" button on both to refresh file contentshttps://gitlab.idiap.ch/beat/beat.editor/-/issues/87Re-enable chrome headless testing support once chrome >55 comes out2018-12-19T19:36:13ZJaden DIEFENBAUGHRe-enable chrome headless testing support once chrome >55 comes outThere's a bug about "lost UI shared context" with chrome/chromium headless that'll probably be solved in the next version. For now just use FF for testing since I haven't ever noticed any test results differences between the two browsers.There's a bug about "lost UI shared context" with chrome/chromium headless that'll probably be solved in the next version. For now just use FF for testing since I haven't ever noticed any test results differences between the two browsers.JS & Python Testinghttps://gitlab.idiap.ch/beat/beat.cmdline/-/issues/24Store prefix on configuration file2018-12-12T09:14:17ZAndré AnjosStore prefix on configuration fileIt would be nice to be able to store the prefix on the configuration file.It would be nice to be able to store the prefix on the configuration file.André AnjosAndré Anjoshttps://gitlab.idiap.ch/beat/beat.core/-/issues/53Test test_scheduler_last (beat.core.test.test_worker.TestConnection) is hanging2018-05-06T01:05:51ZAndré AnjosTest test_scheduler_last (beat.core.test.test_worker.TestConnection) is hangingThis test seems to hang from time to time on the CI. Example:
https://gitlab.idiap.ch/beat/beat.core/-/jobs/133229This test seems to hang from time to time on the CI. Example:
https://gitlab.idiap.ch/beat/beat.core/-/jobs/133229Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/52Cannot run on a Mac with Docker installed2018-12-12T08:05:24ZAndré AnjosCannot run on a Mac with Docker installedThe current test suite works well on a Linux box with the docker daemon installed, but equivalent tests cannot be executed on a MacOS system also with a docker daemon installed.
To reproduce this issue, install Docker for Mac (https://d...The current test suite works well on a Linux box with the docker daemon installed, but equivalent tests cannot be executed on a MacOS system also with a docker daemon installed.
To reproduce this issue, install Docker for Mac (https://docs.docker.com/docker-for-mac/install/#download-docker-for-mac) and try to run the docker test units.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/51Cannot run two CI's in parallel2018-09-18T17:27:09ZAndré AnjosCannot run two CI's in parallelWe're presently facing an issue that, from time to time, tests on the MacOS CI will fail, after hanging for 1h30.
The problem seems to be that some of the tests are probably communicating to other processes using the same network/IO inf...We're presently facing an issue that, from time to time, tests on the MacOS CI will fail, after hanging for 1h30.
The problem seems to be that some of the tests are probably communicating to other processes using the same network/IO infrastructure (e.g. port number).
Here is an example dead-lock:
MacOS + python 27: https://gitlab.idiap.ch/beat/beat.core/-/jobs/133089
MacOS + python 36: https://gitlab.idiap.ch/beat/beat.core/-/jobs/133090Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.cmdline/-/issues/22[CI] issue with docker and zc.buildout2018-04-19T14:08:51ZFlavio TARSETTI[CI] issue with docker and zc.buildoutSeems like the CI is having a strange issue with docker and and zc.buildout (apparently):
https://gitlab.idiap.ch/beat/beat.cmdline/-/jobs/132525
No hurry @samuel.gaist, it's not blocking for me at the moment, it was just to raise your...Seems like the CI is having a strange issue with docker and and zc.buildout (apparently):
https://gitlab.idiap.ch/beat/beat.cmdline/-/jobs/132525
No hurry @samuel.gaist, it's not blocking for me at the moment, it was just to raise your attention on this.
Thanks,
Flaviohttps://gitlab.idiap.ch/beat/beat.web/-/issues/496[supervision] Add notification about supervision requests2020-06-17T10:03:13ZAndré Anjos[supervision] Add notification about supervision requestsCurrently, supervision requests do not show in the event stack of users, but that is a feature which would be nice as that would show up in the user's starter page.Currently, supervision requests do not show in the event stack of users, but that is a feature which would be nice as that would show up in the user's starter page.Flavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/86Update plotterparameter schema & editor2018-04-17T09:23:28ZJaden DIEFENBAUGHUpdate plotterparameter schema & editorflavio's changes in beat/beat.core#49flavio's changes in beat/beat.core#49https://gitlab.idiap.ch/beat/beat.core/-/issues/50Add support for fetching available package info from docker containers2018-10-24T07:39:16ZJaden DIEFENBAUGHAdd support for fetching available package info from docker containersSince (basically?) all the packages in a processing environment are installed through conda, and `docker-py` has support for [getting files from containers](https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.contain...Since (basically?) all the packages in a processing environment are installed through conda, and `docker-py` has support for [getting files from containers](https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.Container.get_archive), getting the conda info from the container will give us the package info.
The format should be a dict of packages names pointing to their versions.https://gitlab.idiap.ch/beat/beat.cmdline/-/issues/21Add local environment querying support2018-05-15T13:53:51ZJaden DIEFENBAUGHAdd local environment querying support(This is the first part of beat.editor#77)
Right now `beat.core` can tell us about the available local environments available (see [`dock.py`](https://gitlab.idiap.ch/beat/beat.core/blob/a8871304860b9e7c87f3b7ec08c0b308ccd0908a/beat/cor...(This is the first part of beat.editor#77)
Right now `beat.core` can tell us about the available local environments available (see [`dock.py`](https://gitlab.idiap.ch/beat/beat.core/blob/a8871304860b9e7c87f3b7ec08c0b308ccd0908a/beat/core/dock.py#L149)). The goal is to give `beat.editor` access to this info, which means that `beat.cmdline` needs a nice way to supply this info.
I'd add a new beat sub-app, `environment` (with an `env` shortcut) with a `list` command that would give the necessary info. Ideally the environment info would be provided in some machine-parseable format and provide at least:
- environment names & versions
- queues for each environment along with their slot counts
- the environments' packages (names & versions)
Right now I'm thinking to just use JSON of the same format as the beat.web endpoint for environments. [BEAT staging example](http://beatweb-staging.lab.idiap.ch:80/api/v1/backend/environments/):
```json
[
{
"name": "Scientific Python 2.7",
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "0.0.4",
"short_description": "Scientific Python 2.7"
},
{
"name": "Scientific Python 2.7",
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "1.0.0",
"short_description": "Scientific Python 2.7"
}
]
```
The one thing missing is the package info. There would be a `"packages"` field added to each environment that points to a dict consisting of package names for keys and versions for values:
```json
[
{
"name": "Scientific Python 2.7",
"packages": {
"alabaster": "0.7.6",
"babel": "1.3",
...
},
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "0.0.4",
"short_description": "Scientific Python 2.7"
},
{
"name": "Scientific Python 2.7",
"packages": {
"alabaster": "0.7.10",
"Babel": "2.4.0",
...
},
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "1.0.0",
"short_description": "Scientific Python 2.7"
}
]
```
This is the same extension proposed in beat.web#495.Jaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/85update to new major/minor versions of deps2018-05-30T14:09:09ZJaden DIEFENBAUGHupdate to new major/minor versions of depsThe API might be changing
- d3
- react-popperThe API might be changing
- d3
- react-popperhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/84Bundle size & Perf optimization2019-01-28T04:36:42ZJaden DIEFENBAUGHBundle size & Perf optimizationMake sure to check out https://github.com/GoogleChromeLabs/webpack-libs-optimizations
Also look into https://preactjs.com/, it's a pretty well known drop-in replacement for react thats not just like 100kB smaller but also way fasterMake sure to check out https://github.com/GoogleChromeLabs/webpack-libs-optimizations
Also look into https://preactjs.com/, it's a pretty well known drop-in replacement for react thats not just like 100kB smaller but also way fasterhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/83BEAT Tutorial2018-05-14T07:45:51ZJaden DIEFENBAUGHBEAT TutorialRough schedule for 3 hour tutorial:
- Introducing BEAT: ~30min
- Ecosystem walkthrough using the Iris dataset: 1hr 15min
- Explaining the experiment and breaking it down conceptually
- Experiment design
- Introduce the ...Rough schedule for 3 hour tutorial:
- Introducing BEAT: ~30min
- Ecosystem walkthrough using the Iris dataset: 1hr 15min
- Explaining the experiment and breaking it down conceptually
- Experiment design
- Introduce the Iris dataset
- Introduce the problem (discriminating across the three species using just measurement samples)
- Propose the used method: LDA
- Breaking down this problem/solution into necessary steps
- Getting the data
- Preprocessing it
- Extracting features
- Training the machine
- Scoring
- Displaying results
- Understanding how/why to split into BEAT blocks
- The MapReduce concept -> blocks & connections
- Synchronization & the "every block is a for-loop" thing
- On-paper toolchain (?)
- Databases
- Quickly introducing Bob
- Creating a database
- Dataformats
- Intro to pulling stuff from platform
- Creating one/some manually
- Algorithms
- What are "algorithms" - analyzer vs normal
- Anatomy of an algorithm file (functions)
- More Bob intro
- Analyzer explanation & create analyzer
- Toolchains
- Using algorithms to create toolchain blocks
- Copying & cool editor stuff
- Discussing toolchain connections & types
- Putting it all together: Experiments
- Explaining how experiments relate to other things
- Creating an experiment and assigning datasets/blocks/parameters
- Plots
- How do plots work? Plotters & plotterparameters
- Creation of plotters
- Creation of plotterparameters
- Generating plots for an experiment
- Pushing back to the platform (?)
- More complex & practical example using AT&T eigenfaces: 30min
- Conceptual breakdown: experiment design
- breaking down eigenface into necessary steps
- understanding how/why to split into BEAT blocks & highlighting reusability stuff
- on-paper toolchain
- toolchain
- Reusing toolchains & other components with DCT-GMM: 15min
- designing using the above stuff
- "wow look reusing stuff"
- Creating a new environment off of an existing one with `beat.env.python`: 30min
- Intro to environments & how that all works
- docker executor
- showing where environments are
- compiling/making an environment
- changing the env & making a new one