beat issueshttps://gitlab.idiap.ch/groups/beat/-/issues2018-12-12T08:05:24Zhttps://gitlab.idiap.ch/beat/beat.core/-/issues/52Cannot run on a Mac with Docker installed2018-12-12T08:05:24ZAndré AnjosCannot run on a Mac with Docker installedThe current test suite works well on a Linux box with the docker daemon installed, but equivalent tests cannot be executed on a MacOS system also with a docker daemon installed.
To reproduce this issue, install Docker for Mac (https://d...The current test suite works well on a Linux box with the docker daemon installed, but equivalent tests cannot be executed on a MacOS system also with a docker daemon installed.
To reproduce this issue, install Docker for Mac (https://docs.docker.com/docker-for-mac/install/#download-docker-for-mac) and try to run the docker test units.Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.core/-/issues/51Cannot run two CI's in parallel2018-09-18T17:27:09ZAndré AnjosCannot run two CI's in parallelWe're presently facing an issue that, from time to time, tests on the MacOS CI will fail, after hanging for 1h30.
The problem seems to be that some of the tests are probably communicating to other processes using the same network/IO inf...We're presently facing an issue that, from time to time, tests on the MacOS CI will fail, after hanging for 1h30.
The problem seems to be that some of the tests are probably communicating to other processes using the same network/IO infrastructure (e.g. port number).
Here is an example dead-lock:
MacOS + python 27: https://gitlab.idiap.ch/beat/beat.core/-/jobs/133089
MacOS + python 36: https://gitlab.idiap.ch/beat/beat.core/-/jobs/133090Samuel GAISTSamuel GAISThttps://gitlab.idiap.ch/beat/beat.cmdline/-/issues/22[CI] issue with docker and zc.buildout2018-04-19T14:08:51ZFlavio TARSETTI[CI] issue with docker and zc.buildoutSeems like the CI is having a strange issue with docker and and zc.buildout (apparently):
https://gitlab.idiap.ch/beat/beat.cmdline/-/jobs/132525
No hurry @samuel.gaist, it's not blocking for me at the moment, it was just to raise your...Seems like the CI is having a strange issue with docker and and zc.buildout (apparently):
https://gitlab.idiap.ch/beat/beat.cmdline/-/jobs/132525
No hurry @samuel.gaist, it's not blocking for me at the moment, it was just to raise your attention on this.
Thanks,
Flaviohttps://gitlab.idiap.ch/beat/beat.web/-/issues/496[supervision] Add notification about supervision requests2020-06-17T10:03:13ZAndré Anjos[supervision] Add notification about supervision requestsCurrently, supervision requests do not show in the event stack of users, but that is a feature which would be nice as that would show up in the user's starter page.Currently, supervision requests do not show in the event stack of users, but that is a feature which would be nice as that would show up in the user's starter page.Flavio TARSETTIFlavio TARSETTIhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/86Update plotterparameter schema & editor2018-04-17T09:23:28ZJaden DIEFENBAUGHUpdate plotterparameter schema & editorflavio's changes in beat/beat.core#49flavio's changes in beat/beat.core#49https://gitlab.idiap.ch/beat/beat.core/-/issues/50Add support for fetching available package info from docker containers2018-10-24T07:39:16ZJaden DIEFENBAUGHAdd support for fetching available package info from docker containersSince (basically?) all the packages in a processing environment are installed through conda, and `docker-py` has support for [getting files from containers](https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.contain...Since (basically?) all the packages in a processing environment are installed through conda, and `docker-py` has support for [getting files from containers](https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.Container.get_archive), getting the conda info from the container will give us the package info.
The format should be a dict of packages names pointing to their versions.https://gitlab.idiap.ch/beat/beat.cmdline/-/issues/21Add local environment querying support2018-05-15T13:53:51ZJaden DIEFENBAUGHAdd local environment querying support(This is the first part of beat.editor#77)
Right now `beat.core` can tell us about the available local environments available (see [`dock.py`](https://gitlab.idiap.ch/beat/beat.core/blob/a8871304860b9e7c87f3b7ec08c0b308ccd0908a/beat/cor...(This is the first part of beat.editor#77)
Right now `beat.core` can tell us about the available local environments available (see [`dock.py`](https://gitlab.idiap.ch/beat/beat.core/blob/a8871304860b9e7c87f3b7ec08c0b308ccd0908a/beat/core/dock.py#L149)). The goal is to give `beat.editor` access to this info, which means that `beat.cmdline` needs a nice way to supply this info.
I'd add a new beat sub-app, `environment` (with an `env` shortcut) with a `list` command that would give the necessary info. Ideally the environment info would be provided in some machine-parseable format and provide at least:
- environment names & versions
- queues for each environment along with their slot counts
- the environments' packages (names & versions)
Right now I'm thinking to just use JSON of the same format as the beat.web endpoint for environments. [BEAT staging example](http://beatweb-staging.lab.idiap.ch:80/api/v1/backend/environments/):
```json
[
{
"name": "Scientific Python 2.7",
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "0.0.4",
"short_description": "Scientific Python 2.7"
},
{
"name": "Scientific Python 2.7",
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "1.0.0",
"short_description": "Scientific Python 2.7"
}
]
```
The one thing missing is the package info. There would be a `"packages"` field added to each environment that points to a dict consisting of package names for keys and versions for values:
```json
[
{
"name": "Scientific Python 2.7",
"packages": {
"alabaster": "0.7.6",
"babel": "1.3",
...
},
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "0.0.4",
"short_description": "Scientific Python 2.7"
},
{
"name": "Scientific Python 2.7",
"packages": {
"alabaster": "0.7.10",
"Babel": "2.4.0",
...
},
"queues": {
"Default": {
"memory_limit": 5983,
"nb_slots": 2,
"max_slots_per_user": 2,
"nb_cores_per_slot": 1,
"time_limit": 360
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "1.0.0",
"short_description": "Scientific Python 2.7"
}
]
```
This is the same extension proposed in beat.web#495.Jaden DIEFENBAUGHJaden DIEFENBAUGHhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/85update to new major/minor versions of deps2018-05-30T14:09:09ZJaden DIEFENBAUGHupdate to new major/minor versions of depsThe API might be changing
- d3
- react-popperThe API might be changing
- d3
- react-popperhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/84Bundle size & Perf optimization2019-01-28T04:36:42ZJaden DIEFENBAUGHBundle size & Perf optimizationMake sure to check out https://github.com/GoogleChromeLabs/webpack-libs-optimizations
Also look into https://preactjs.com/, it's a pretty well known drop-in replacement for react thats not just like 100kB smaller but also way fasterMake sure to check out https://github.com/GoogleChromeLabs/webpack-libs-optimizations
Also look into https://preactjs.com/, it's a pretty well known drop-in replacement for react thats not just like 100kB smaller but also way fasterhttps://gitlab.idiap.ch/beat/beat.core/-/issues/49Update the plotterparameter schema to associate it to a plotter2018-04-17T07:55:49ZJaden DIEFENBAUGHUpdate the plotterparameter schema to associate it to a plotterRight now there's no way to tell via the naming scheme or the plotterparameter JSON which plotter the parameter is associated to. The easiest way would be to add a field, say "plotter", which has the associated plotter object's name. See...Right now there's no way to tell via the naming scheme or the plotterparameter JSON which plotter the parameter is associated to. The easiest way would be to add a field, say "plotter", which has the associated plotter object's name. See beat.cmdline#20 for the discussion that led up to this.
The idea could be something like what I suggested in the other thread:
```json
{
"plotter": "plot/bar/1",
"grid": true,
"title": "DET ISO/IEC 19795-1:2006",
"title-fontsize": 12,
"det": true,
"legend-loc": "best",
"width": 800,
"height": 600,
"dpi": 120,
"legend-fontsize": 8,
"xlim-left": 0,
"xlim-right": 40,
"ylim-bottom": 0,
"ylim-top": 40,
"axis-fontsize": 10
}
```
The "plotter" field stores the reference to the associated plotter.
However, it might be best to use a field name that the user couldn't accidentally use, such as `#plotter`.
@samuel.gaist @flavio.tarsetti @andre.anjos what do you all think? This is sorta blocking time-sensitive work so it'd be better to decide quickly.https://gitlab.idiap.ch/beat/beat.editor/-/issues/83BEAT Tutorial2018-05-14T07:45:51ZJaden DIEFENBAUGHBEAT TutorialRough schedule for 3 hour tutorial:
- Introducing BEAT: ~30min
- Ecosystem walkthrough using the Iris dataset: 1hr 15min
- Explaining the experiment and breaking it down conceptually
- Experiment design
- Introduce the ...Rough schedule for 3 hour tutorial:
- Introducing BEAT: ~30min
- Ecosystem walkthrough using the Iris dataset: 1hr 15min
- Explaining the experiment and breaking it down conceptually
- Experiment design
- Introduce the Iris dataset
- Introduce the problem (discriminating across the three species using just measurement samples)
- Propose the used method: LDA
- Breaking down this problem/solution into necessary steps
- Getting the data
- Preprocessing it
- Extracting features
- Training the machine
- Scoring
- Displaying results
- Understanding how/why to split into BEAT blocks
- The MapReduce concept -> blocks & connections
- Synchronization & the "every block is a for-loop" thing
- On-paper toolchain (?)
- Databases
- Quickly introducing Bob
- Creating a database
- Dataformats
- Intro to pulling stuff from platform
- Creating one/some manually
- Algorithms
- What are "algorithms" - analyzer vs normal
- Anatomy of an algorithm file (functions)
- More Bob intro
- Analyzer explanation & create analyzer
- Toolchains
- Using algorithms to create toolchain blocks
- Copying & cool editor stuff
- Discussing toolchain connections & types
- Putting it all together: Experiments
- Explaining how experiments relate to other things
- Creating an experiment and assigning datasets/blocks/parameters
- Plots
- How do plots work? Plotters & plotterparameters
- Creation of plotters
- Creation of plotterparameters
- Generating plots for an experiment
- Pushing back to the platform (?)
- More complex & practical example using AT&T eigenfaces: 30min
- Conceptual breakdown: experiment design
- breaking down eigenface into necessary steps
- understanding how/why to split into BEAT blocks & highlighting reusability stuff
- on-paper toolchain
- toolchain
- Reusing toolchains & other components with DCT-GMM: 15min
- designing using the above stuff
- "wow look reusing stuff"
- Creating a new environment off of an existing one with `beat.env.python`: 30min
- Intro to environments & how that all works
- docker executor
- showing where environments are
- compiling/making an environment
- changing the env & making a new onehttps://gitlab.idiap.ch/beat/beat.editor/-/issues/82Plotter & Plotterparameter editors2018-04-11T13:44:42ZJaden DIEFENBAUGHPlotter & Plotterparameter editorshttps://gitlab.idiap.ch/beat/beat.cmdline/-/issues/20Format for storing plotterparameters in prefix2018-05-15T13:54:13ZJaden DIEFENBAUGHFormat for storing plotterparameters in prefix@flavio.tarsetti how does the system associate plotters & plotterparameters?
The names aren't associated, as the plotterparameter `plot/det_40x40/1` isn't a similar name to `plot/isoroc/1`. And the plotterparameter content is:
```json
...@flavio.tarsetti how does the system associate plotters & plotterparameters?
The names aren't associated, as the plotterparameter `plot/det_40x40/1` isn't a similar name to `plot/isoroc/1`. And the plotterparameter content is:
```json
{
"grid": true,
"title": "DET ISO/IEC 19795-1:2006",
"title-fontsize": 12,
"det": true,
"legend-loc": "best",
"width": 800,
"height": 600,
"dpi": 120,
"legend-fontsize": 8,
"xlim-left": 0,
"xlim-right": 40,
"ylim-bottom": 0,
"ylim-top": 40,
"axis-fontsize": 10
}
```
How do we associate them?https://gitlab.idiap.ch/beat/beat.editor/-/issues/81Make exp using Tensorflow/PyTorch to use a DNN for M-NIST digit classification2018-04-16T15:20:12ZJaden DIEFENBAUGHMake exp using Tensorflow/PyTorch to use a DNN for M-NIST digit classificationSame as other conversion tasks, but this time doing something much differentSame as other conversion tasks, but this time doing something much differenthttps://gitlab.idiap.ch/beat/beat.editor/-/issues/80Convert a DCT-GMM-based face verification Bob experiment to BEAT2018-05-15T09:52:23ZJaden DIEFENBAUGHConvert a DCT-GMM-based face verification Bob experiment to BEATAfter #58, try to use the same toolchain (maybe the same database?) to implement a similar but different experiementAfter #58, try to use the same toolchain (maybe the same database?) to implement a similar but different experiementhttps://gitlab.idiap.ch/beat/beat.editor/-/issues/78algs by default should not be analyzers2018-03-29T08:19:55ZJaden DIEFENBAUGHalgs by default should not be analyzershttps://gitlab.idiap.ch/beat/beat.cmdline/-/issues/19Fetch environment data from beat.web instances2018-04-13T12:39:53ZJaden DIEFENBAUGHFetch environment data from beat.web instances...using the `/api/v1/backend/environments/` endpoint.
Not sure how this would be stored...probably just dump the JSON in an `environments.json` file in the `.beat` user folder.
Needed for the June 2018 release!...using the `/api/v1/backend/environments/` endpoint.
Not sure how this would be stored...probably just dump the JSON in an `environments.json` file in the `.beat` user folder.
Needed for the June 2018 release!https://gitlab.idiap.ch/beat/beat.web/-/issues/495Expose more info about the available environments from the REST API2020-03-30T15:38:45ZJaden DIEFENBAUGHExpose more info about the available environments from the REST APIRight now we get something like this:
```json
[
{
"name": "Cxx backend",
"queues": {
"queue": {
"memory_limit": 1985,
"nb_slots": 0,
"max_slots_per_user": 8,
"nb_cores_per_slot": 1,
...Right now we get something like this:
```json
[
{
"name": "Cxx backend",
"queues": {
"queue": {
"memory_limit": 1985,
"nb_slots": 0,
"max_slots_per_user": 8,
"nb_cores_per_slot": 1,
"time_limit": 1440
}
},
"accessibility": "public",
"languages": [
"cxx"
],
"version": "1.1.0",
"short_description": "C++ backend"
},
{
"name": "Python 2.7",
"queues": {
"queue": {
"memory_limit": 1985,
"nb_slots": 0,
"max_slots_per_user": 8,
"nb_cores_per_slot": 1,
"time_limit": 1440
}
},
"accessibility": "public",
"languages": [
"python"
],
"version": "1.2.0",
"short_description": "Local python interpreter"
}
]
```
We need to add a "modules" field to the environments that lists packages & versions that are available. It would look something like this:
```json
[
...
{
"name": "Python 2.7",
...
"packages": {
"alabaster": "0.7.10",
"Babel": "2.4.0",
...
},
"version": "1.2.0",
"short_description": "Local python interpreter"
}
]
```
This is needed for the June 2018 deliverable!https://gitlab.idiap.ch/beat/beat.editor/-/issues/76TODOs for June Deliverable2018-05-30T14:08:33ZJaden DIEFENBAUGHTODOs for June DeliverableMilestones:
| Date | Milestone |
| :--- | :-------- |
| 07.06 | Tutorial date |
| 28.05 | VM ready for tutorial (production) |
| 28.05 | Deployable system / All software is released (tagged) for HES-SO tutorial |
| 21.05 | Fix bugs & re...Milestones:
| Date | Milestone |
| :--- | :-------- |
| 07.06 | Tutorial date |
| 28.05 | VM ready for tutorial (production) |
| 28.05 | Deployable system / All software is released (tagged) for HES-SO tutorial |
| 21.05 | Fix bugs & rewrite tutorial |
| 09.05 | Tutorial at Idiap with Conda packages (beta) |
| 07.05 | First version of deployable system based on Conda (all packages) |
| 23.04 | Feature freeze |
| 04.04 | Encode DCT-GMM-based Face verification (#80) |
| 28.03 | Encode AT&T-based Eigenfaces (from bob.bio.base) (#58) |https://gitlab.idiap.ch/beat/beat.editor/-/issues/75replace import statements in templates with note on how imports in algs/dbs/l...2018-04-09T08:44:12ZJaden DIEFENBAUGHreplace import statements in templates with note on how imports in algs/dbs/libs work