Commit cccd2a14 authored by Amir MOHAMMADI's avatar Amir MOHAMMADI

Add beat docs here

parent 2420728c
Pipeline #24351 passed with stages
in 4 minutes and 41 seconds
......@@ -80,6 +80,7 @@ deploy_beta:
environment: beta
only:
- master
- new
deploy_stable:
......
......@@ -12,16 +12,16 @@ for pkg in `cat requirements.txt | sed -e '/^\s*#.*/d;/^\s*$/d'`; do
version=1.4.x
fi
if [[ $pkg == "beat.core" ]]; then
version=1.6.x
version=master
fi
if [[ $pkg == "beat.backend.python" ]]; then
version=1.5.x
version=master
fi
if [[ $pkg == "beat.web" ]]; then
version=1.4.x
fi
if [[ $pkg == "beat.editor" ]]; then
version=docs
version=master
fi
git clone --depth 1 -b $version ${GITLAB_CHECKOUT_STRATEGY}beat/${pkg}.git doc/$pkg || \
{ git -C doc/$pkg reset --hard HEAD && \
......
......@@ -17,7 +17,6 @@ requirements:
- beat.cmdline
- beat.backend.python
- beat.editor
- bob.extension
test:
commands:
......@@ -28,6 +27,7 @@ test:
- sphinx
- sphinx_rtd_theme
- matplotlib
- bob.extension
about:
home: https://www.idiap.ch/software/beat/
......
This diff is collapsed.
.. vim: set fileencoding=utf-8 :
.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ ..
.. Contact: beat.support@idiap.ch ..
.. ..
.. This file is part of the beat.core module of the BEAT platform. ..
.. ..
.. Commercial License Usage ..
.. Licensees holding valid commercial BEAT licenses may use this file in ..
.. accordance with the terms contained in a written agreement between you ..
.. and Idiap. For further information contact tto@idiap.ch ..
.. ..
.. Alternatively, this file may be used under the terms of the GNU Affero ..
.. Public License version 3 as published by the Free Software and appearing ..
.. in the file LICENSE.AGPL included in the packaging of this file. ..
.. The BEAT platform is distributed in the hope that it will be useful, but ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE. ..
.. ..
.. You should have received a copy of the GNU Affero Public License along ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/. ..
==========
Databases
==========
A database is a collection of data files, one for each output of the database.
This data are inputs to the BEAT toolchains. Therefore, it is important to
define evaluation protocols, which describe how a specific system must use the
raw data of a given database.
For instance, a recognition system will typically use a subset of the data to
train a recognition `model`, while another subset of data will be used to
evaluate the performance of this model.
Structure of a database
-----------------------
A database has the following structure on disk::
database_name/
output1_name.data
output2_name.data
...
outputN_name.data
For a given database, the BEAT platform will typically stores information
about the root folder containing this raw data as well as a description of
it.
Evaluation protocols
--------------------
A BEAT evaluation protocol consists of several ``datasets``, each datasets
having several ``outputs`` with well-defined data formats. In practice,
each dataset will typically be used for a different purpose.
For instance, in the case of a simple face recognition protocol, the
database may be split into three datasets: one for training, one for enrolling
client-specific model, and one for testing these models.
The training dataset may have two outputs: grayscale images as two-dimensional
array of type `uint8` and client id as `uint64` integers.
The BEAT platform is data-driven, which means that all the outputs of a given
dataset are synchronized. The way the data is generated by each template
is defined in a piece of code called the ``database view``. It is important
that a database view has a deterministic behavior for reproducibility
purposes.
Database set templates
----------------------
In practice, different databases used for the same purpose may have the exact
same datasets with the exact same outputs (and attached data formats). In this
case, it is interesting to abstract the definition of the database sets from
a given database. BEAT defines ``database set templates`` for this purpose.
For instance, the simple face recognition evaluation protocol described above,
which consists of three datasets and few inputs may be abstracted in a
database set template. This template defines both the datasets, their outputs
as well as their corresponding data formats. Next, if several databases
implements such a protocol, they may rely on the same `database set template`.
Similarly, evaluation protocols testing different conditions (such as
enrolling on clean and testing on clean data vs. enrolling on clean and
testing on noisy data) may rely on the same database set template.
In practice, this reduces the amount of work to integrate new databases and/or
new evaluation protocols into the platform. Besides, at the experiment level,
this allows to re-use a toolchain on a different database, with almost no
configuration changes from the user.
This diff is collapsed.
.. vim: set fileencoding=utf-8 :
.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ ..
.. Contact: beat.support@idiap.ch ..
.. ..
.. This file is part of the beat.core module of the BEAT platform. ..
.. ..
.. Commercial License Usage ..
.. Licensees holding valid commercial BEAT licenses may use this file in ..
.. accordance with the terms contained in a written agreement between you ..
.. and Idiap. For further information contact tto@idiap.ch ..
.. ..
.. Alternatively, this file may be used under the terms of the GNU Affero ..
.. Public License version 3 as published by the Free Software and appearing ..
.. in the file LICENSE.AGPL included in the packaging of this file. ..
.. The BEAT platform is distributed in the hope that it will be useful, but ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE. ..
.. ..
.. You should have received a copy of the GNU Affero Public License along ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/. ..
.. _beat-core-experiments:
============
Experiments
============
An experiment is the reunion of algorithms, datasets, a toolchain and
parameters that allow the platform to schedule and run the prescribed recipe
to produce displayable results. Defining a BEAT experiment can be seen as
configuring the processing blocks of a toolchain, such as selecting which
database, algorithms and algorithm parameters to use.
.. _beat-core-experiments-declaration:
Declaration of an experiment
----------------------------
.. note::
One needs only to declare an experiment using those specifications when not
using the web interface (i.e. when doing local development or using the web
api). The web interface provides a user-friendly way to configure an
experiment.
An experiment is declared in a JSON file, and must contain at least the following
fields:
.. code-block:: javascript
{
"datasets": [
],
"blocks": [
],
"analyzers": [
],
"globals": [
]
}
.. _beat-core-experiments-datasets:
Declaration of the dataset(s)
-----------------------------
The dataset inputs are defined by the toolchain. However, the toolchain does
not describe which data to plug in each dataset input.
This is the role of the field `datasets` from an experiment.
For each dataset, an experiment must specify three attributes as follows:
.. code-block:: javascript
{
"datasets": [
"templates": {
"set": "templates",
"protocol": "idiap",
"database": "atnt/1"
},
...
],
...
}
The key of an experiment dataset must correspond to the desired dataset name
from the toolchain. Then, three fields must be given:
* `database`: the database name and version
* `protocol`: the protocol name
* `set`: the dataset name of this database to associate to this toolchain
dataset
.. _beat-core-experiments-blocks:
Declaration of the block(s)
---------------------------
The blocks are defined by the toolchain. However, the toolchain does not
describe which algorithm to run in each processing block, and how each of these
algorithms are parametrized.
This is the role of the field `blocks` from an experiment.
For each block, an experiment must specify four attributes as follows:
.. code-block:: javascript
{
"blocks": {
"linear_machine_training": {
"inputs": {
"image": "image"
},
"parameters": {},
"algorithm": "tutorial/pca/1",
"outputs": {
"subspace": "subspace"
}
},
...
},
...
}
The key of an experiment block must correspond to the desired block from the
toolchain. Then, four fields must be given:
* `algorithm`: the algorithm to use (author_name/algorithm_name/version)
* `inputs`: the list of inputs. The key is the algorithm input, while the
value is the corresponding toolchain input.
* `outputs`: the list of outputs. The key is the algorithm output, while the
value is the corresponding toolchain output.
* `parameters`: the algorithm parameters to use for this processing block
.. note::
When setting an algorithm in a processing block, this will also set the
dataformats of the outputs (and inputs) of this block. In particular,
this has an impact on all the inputs of blocks connected to those outputs,
which must have the same data formats (or be an extension of these data
formats). The platform automatically validate that the data formats of
consecutive blocks are compatible.
.. _beat-core-experiments-analyzers:
Declaration of the analyzer(s)
------------------------------
Analyzers are similar to algorithms, except that they run on toolchain
endpoints. There configuration is very similar to the one of regular blocks,
except that they have no `outputs`:
.. code-block:: javascript
{
"analyzers": {
"analysis": {
"inputs": {
"scores": "scores"
},
"algorithm": "tutorial/postperf/1"
}
},
}
Global parameters
-----------------
Each block and analyzer may rely on its own local parameters. However, several
blocks may rely on the exact same parameters. In this case, it is more
convenient to define those globally.
For an experiment, this is achieved using the `globals` field in its JSON
declaration. For instance:
.. code-block:: javascript
{
"globals": {
"queue": "Default",
"environment": {
"version": "0.0.3",
"name": "Scientific Python 2.7"
},
"tutorial/pca/1": {
"number-of-components": "5"
}
},
...
}
This source diff could not be displayed because it is too large. You can view the blob instead.
# use mscgen to create a diagram from this representation:
# $ mscgen -T png -o img/execute.png -i img/execute.msc
msc {
hscale = "2.0";
io [label="BEAT Infrastructure (language agnostic)"], up [label="Execute (user code)"];
up->io [ label = "nxt channel" ];
io->up [ label = "2 name1 <bin1> name2 <bin2>" ];
up->io [ label = "hmd channel name" ];
io->up [ label = "tru" ];
up->io [ label = "oic name" ];
io->up [ label = "tru" ];
up->io [ label = "wrt out sz <bin>" ];
io->up [ label = "ack" ];
...;
up->io [ label = "hmd channel name" ];
io->up [ label = "fal" ];
up->io [ label = "don" ];
...;
io->up [ label = "ack" ];
}
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1"
width="1200px" height="392px"
viewBox="0 0 1200 392"
xmlns="http://www.w3.org/2000/svg" shape-rendering="crispEdges"
stroke-width="1" text-rendering="geometricPrecision">
<polygon fill="white" points="191,7 407,7 407,16 191,16"/>
<text x="300" y="16" textLength="214" font-family="Helvetica" font-size="12" fill="black" text-anchor="middle">
BEAT Infrastructure (language agnostic)
</text>
<polygon fill="white" points="845,7 953,7 953,16 845,16"/>
<text x="900" y="16" textLength="107" font-family="Helvetica" font-size="12" fill="black" text-anchor="middle">
Execute (user code)
</text>
<line x1="300" y1="22" x2="300" y2="50" stroke="black"/>
<line x1="900" y1="22" x2="900" y2="50" stroke="black"/>
<line x1="900" y1="33" x2="300" y2="33" stroke="black"/>
<line x1="300" y1="33" x2="310" y2="39" stroke="black"/>
<polygon fill="white" points="568,23 631,23 631,32 568,32"/>
<text x="569" y="32" textLength="61" font-family="Helvetica" font-size="12" fill="black">
nxt channel
</text>
<line x1="300" y1="50" x2="300" y2="78" stroke="black"/>
<line x1="900" y1="50" x2="900" y2="78" stroke="black"/>
<line x1="300" y1="61" x2="900" y2="61" stroke="black"/>
<line x1="900" y1="61" x2="890" y2="67" stroke="black"/>
<polygon fill="white" points="515,51 684,51 684,60 515,60"/>
<text x="516" y="60" textLength="167" font-family="Helvetica" font-size="12" fill="black">
2 name1 &lt;bin1&gt; name2 &lt;bin2&gt;
</text>
<line x1="300" y1="78" x2="300" y2="106" stroke="black"/>
<line x1="900" y1="78" x2="900" y2="106" stroke="black"/>
<line x1="900" y1="89" x2="300" y2="89" stroke="black"/>
<line x1="300" y1="89" x2="310" y2="95" stroke="black"/>
<polygon fill="white" points="547,79 651,79 651,88 547,88"/>
<text x="548" y="88" textLength="102" font-family="Helvetica" font-size="12" fill="black">
hmd channel name
</text>
<line x1="300" y1="106" x2="300" y2="134" stroke="black"/>
<line x1="900" y1="106" x2="900" y2="134" stroke="black"/>
<line x1="300" y1="117" x2="900" y2="117" stroke="black"/>
<line x1="900" y1="117" x2="890" y2="123" stroke="black"/>
<polygon fill="white" points="591,107 607,107 607,116 591,116"/>
<text x="592" y="116" textLength="14" font-family="Helvetica" font-size="12" fill="black">
tru
</text>
<line x1="300" y1="134" x2="300" y2="162" stroke="black"/>
<line x1="900" y1="134" x2="900" y2="162" stroke="black"/>
<line x1="900" y1="145" x2="300" y2="145" stroke="black"/>
<line x1="300" y1="145" x2="310" y2="151" stroke="black"/>
<polygon fill="white" points="574,135 625,135 625,144 574,144"/>
<text x="575" y="144" textLength="49" font-family="Helvetica" font-size="12" fill="black">
oic name
</text>
<line x1="300" y1="162" x2="300" y2="190" stroke="black"/>
<line x1="900" y1="162" x2="900" y2="190" stroke="black"/>
<line x1="300" y1="173" x2="900" y2="173" stroke="black"/>
<line x1="900" y1="173" x2="890" y2="179" stroke="black"/>
<polygon fill="white" points="591,163 607,163 607,172 591,172"/>
<text x="592" y="172" textLength="14" font-family="Helvetica" font-size="12" fill="black">
tru
</text>
<line x1="300" y1="190" x2="300" y2="218" stroke="black"/>
<line x1="900" y1="190" x2="900" y2="218" stroke="black"/>
<line x1="900" y1="201" x2="300" y2="201" stroke="black"/>
<line x1="300" y1="201" x2="310" y2="207" stroke="black"/>
<polygon fill="white" points="556,191 643,191 643,200 556,200"/>
<text x="557" y="200" textLength="85" font-family="Helvetica" font-size="12" fill="black">
wrt out sz &lt;bin&gt;
</text>
<line x1="300" y1="218" x2="300" y2="246" stroke="black"/>
<line x1="900" y1="218" x2="900" y2="246" stroke="black"/>
<line x1="300" y1="229" x2="900" y2="229" stroke="black"/>
<line x1="900" y1="229" x2="890" y2="235" stroke="black"/>
<polygon fill="white" points="589,219 610,219 610,228 589,228"/>
<text x="590" y="228" textLength="19" font-family="Helvetica" font-size="12" fill="black">
ack
</text>
<line x1="300" y1="246" x2="300" y2="263" stroke="black" stroke-dasharray="2,2"/>
<line x1="900" y1="246" x2="900" y2="263" stroke="black" stroke-dasharray="2,2"/>
<line x1="300" y1="263" x2="300" y2="291" stroke="black"/>
<line x1="900" y1="263" x2="900" y2="291" stroke="black"/>
<line x1="900" y1="274" x2="300" y2="274" stroke="black"/>
<line x1="300" y1="274" x2="310" y2="280" stroke="black"/>
<polygon fill="white" points="547,264 651,264 651,273 547,273"/>
<text x="548" y="273" textLength="102" font-family="Helvetica" font-size="12" fill="black">
hmd channel name
</text>
<line x1="300" y1="291" x2="300" y2="319" stroke="black"/>
<line x1="900" y1="291" x2="900" y2="319" stroke="black"/>
<line x1="300" y1="302" x2="900" y2="302" stroke="black"/>
<line x1="900" y1="302" x2="890" y2="308" stroke="black"/>
<polygon fill="white" points="592,292 607,292 607,301 592,301"/>
<text x="593" y="301" textLength="13" font-family="Helvetica" font-size="12" fill="black">
fal
</text>
<line x1="300" y1="319" x2="300" y2="347" stroke="black"/>
<line x1="900" y1="319" x2="900" y2="347" stroke="black"/>
<line x1="900" y1="330" x2="300" y2="330" stroke="black"/>
<line x1="300" y1="330" x2="310" y2="336" stroke="black"/>
<polygon fill="white" points="588,320 610,320 610,329 588,329"/>
<text x="589" y="329" textLength="20" font-family="Helvetica" font-size="12" fill="black">
don
</text>
<line x1="300" y1="347" x2="300" y2="364" stroke="black" stroke-dasharray="2,2"/>
<line x1="900" y1="347" x2="900" y2="364" stroke="black" stroke-dasharray="2,2"/>
<line x1="300" y1="364" x2="300" y2="392" stroke="black"/>
<line x1="900" y1="364" x2="900" y2="392" stroke="black"/>
<line x1="300" y1="375" x2="900" y2="375" stroke="black"/>
<line x1="900" y1="375" x2="890" y2="381" stroke="black"/>
<polygon fill="white" points="589,365 610,365 610,374 589,374"/>
<text x="590" y="374" textLength="19" font-family="Helvetica" font-size="12" fill="black">
ack
</text>
<line x1="300" y1="386" x2="300" y2="392" stroke="black"/>
<line x1="900" y1="386" x2="900" y2="392" stroke="black"/>
</svg>
digraph "user/triangle/1" {
graph [compound=true rankdir=LR splines=polyline]
subgraph dataset_cluster {
graph [label=datasets rank=same]
set [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><font color="#000000"><b><u>set</u></b><br/>simple/1<br/><i>protocol:set</i></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set2 [label=<<table border="0" cellspacing="0" bgcolor="#e2ffc7"><tr><td><font color="#000000"><b><u>set2</u></b><br/>simple/1<br/><i>protocol:set2</i></font></td><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="output_out" bgcolor="#6AA84F" border="1"><font color="#000000">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
}
echo1 [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr></table></td><td><font color="#000000"><b><u>echo1</u></b><br/>user/integers_echo/1<br/><i>@environment(1) x 1</i></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set:output_out -> echo1:input_in [label="beat/integer/1" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
echo2 [label=<<table border="0" cellspacing="0" bgcolor="#e2ffc7"><tr><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="input_in" bgcolor="#6AA84F" border="1"><font color="#000000">in</font></td></tr></table></td><td><font color="#000000"><b><u>echo2</u></b><br/>user/integers_echo/1<br/><i>@environment(1) x 1</i></font></td><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="output_out" bgcolor="#6AA84F" border="1"><font color="#000000">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set2:output_out -> echo2:input_in [label="beat/integer/1" color="#6AA84F" fontcolor="#6AA84F" fontname=Helvetica fontsize=12]
echo3 [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr><tr><td port="input_in2" bgcolor="#0000FF" border="1"><font color="#ffffff">in2</font></td></tr></table></td><td><font color="#000000"><b><u>echo3</u></b><br/>user/integers_echo_ignore/1<br/><i>@environment(1) x 1</i></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
echo1:output_out -> echo3:input_in [label="beat/integer/1" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
echo2:output_out -> echo3:input_in2 [label="beat/integer/1" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
analysis [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr></table></td><td><font color="#000000"><b><u>analysis</u></b><br/>user/integers_echo_analyzer/1<br/><i>@environment(1) x 1</i></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td bgcolor="#0000FF" border="1"><font color="#ffffff">out_data<br/>(int32)</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
echo3:output_out -> analysis:input_in [label="beat/integer/1" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
subgraph analyzer_cluster {
graph [label=analyzers rank=same]
}
}
digraph "user/triangle/1" {
graph [compound=true rankdir=LR splines=polyline]
subgraph dataset_cluster {
graph [label=datasets rank=same]
set [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><font color="#000000"><b><u>set</u></b></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set2 [label=<<table border="0" cellspacing="0" bgcolor="#e2ffc7"><tr><td><font color="#000000"><b><u>set2</u></b></font></td><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="output_out" bgcolor="#6AA84F" border="1"><font color="#000000">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
}
echo1 [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr></table></td><td><font color="#000000"><b><u>echo1</u></b></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set:output_out -> echo1:input_in [label="" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
echo2 [label=<<table border="0" cellspacing="0" bgcolor="#e2ffc7"><tr><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="input_in" bgcolor="#6AA84F" border="1"><font color="#000000">in</font></td></tr></table></td><td><font color="#000000"><b><u>echo2</u></b></font></td><td><table border="0" cellspacing="5" bgcolor="#e2ffc7"><tr><td port="output_out" bgcolor="#6AA84F" border="1"><font color="#000000">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
set2:output_out -> echo2:input_in [label="" color="#6AA84F" fontcolor="#6AA84F" fontname=Helvetica fontsize=12]
echo3 [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr><tr><td port="input_in2" bgcolor="#0000FF" border="1"><font color="#ffffff">in2</font></td></tr></table></td><td><font color="#000000"><b><u>echo3</u></b></font></td><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="output_out" bgcolor="#0000FF" border="1"><font color="#ffffff">out</font></td></tr></table></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
echo1:output_out -> echo3:input_in [label="" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
echo2:output_out -> echo3:input_in2 [label="" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
analysis [label=<<table border="0" cellspacing="0" bgcolor="#7878ff"><tr><td><table border="0" cellspacing="5" bgcolor="#7878ff"><tr><td port="input_in" bgcolor="#0000FF" border="1"><font color="#ffffff">in</font></td></tr></table></td><td><font color="#000000"><b><u>analysis</u></b></font></td></tr></table>> fontname=Helvetica fontsize=12 shape=none]
echo3:output_out -> analysis:input_in [label="" color="#0000FF" fontcolor="#0000FF" fontname=Helvetica fontsize=12]
subgraph analyzer_cluster {
graph [label=analyzers rank=same]
}
}
.. vim: set fileencoding=utf-8 :
.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ ..
.. Contact: beat.support@idiap.ch ..
.. ..
.. This file is part of the beat.core module of the BEAT platform. ..
.. ..
.. Commercial License Usage ..
.. Licensees holding valid commercial BEAT licenses may use this file in ..
.. accordance with the terms contained in a written agreement between you ..
.. and Idiap. For further information contact tto@idiap.ch ..
.. ..
.. Alternatively, this file may be used under the terms of the GNU Affero ..
.. Public License version 3 as published by the Free Software and appearing ..
.. in the file LICENSE.AGPL included in the packaging of this file. ..
.. The BEAT platform is distributed in the hope that it will be useful, but ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE. ..
.. ..
.. You should have received a copy of the GNU Affero Public License along ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/. ..
.. _beat:
======
BEAT
======
This user guide contains information about BEAT core components, defining
experiments, toolchains and user algorithms among others.
.. toctree::
introduction
dataformats
algorithms
libraries
toolchains
experiments
databases
io
backend_api
develop
api
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
This diff is collapsed.
.. vim: set fileencoding=utf-8 :
.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ ..
.. Contact: beat.support@idiap.ch ..
.. ..
.. This file is part of the beat.core module of the BEAT platform. ..
.. ..
.. Commercial License Usage ..
.. Licensees holding valid commercial BEAT licenses may use this file in ..
.. accordance with the terms contained in a written agreement between you ..
.. and Idiap. For further information contact tto@idiap.ch ..
.. ..
.. Alternatively, this file may be used under the terms of the GNU Affero ..
.. Public License version 3 as published by the Free Software and appearing ..
.. in the file LICENSE.AGPL included in the packaging of this file. ..
.. The BEAT platform is distributed in the hope that it will be useful, but ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE. ..
.. ..
.. You should have received a copy of the GNU Affero Public License along ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/. ..
.. _developerguide-io:
===============
Inputs/Outputs
===============
.. _developerguide-io-introduction:
Introduction
------------
The requirements for the platform when reading/writing data are:
* Ability to manage large and complex data
* Portability to allow the use of heterogeneous environments
Based on our experience and on these requirements, we investigated
the use of HDF5. Unfortunately, HDF5 is not convenient to handle
structures such as arrays of variable-size elements, for instance,
array of strings.
Therefore, we decided to rely on our own binary format.
.. _developerguide-io-strategy:
Binary Format
-------------
Our binary format does *not* contains information about the format of the data
itself, and it is hence necessary to know this format a priori. This means that
the format cannot be inferred from the content of a file.
We rely on the following fundamental C-style formats:
* int8
* int16
* int32
* int64
* uint8
* uint16
* uint32
* uint64
* float32
* float64
* complex64 (first real value, and then imaginary value)
* complex128 (first real value, and then imaginary value)
* bool (written as a byte)
* string