Commit 8e5b65fe authored by Zohreh MOSTAANI's avatar Zohreh MOSTAANI

[core][doc] modifed organization of backend and developer guide

parent 0803f370
......@@ -34,11 +34,70 @@ with a hybrid set of algorithms that execute on different backends. Each
backend can be implemented in a different programming language and contain any
number of (pre-installed) libraries users can call on their algorithms.
This document describes the API required by such backend implementations. The
The requirements for BEAT when reading/writing data are:
* Ability to manage large and complex data
* Portability to allow the use of heterogeneous environments
Based on our experience and on these requirements, we investigated
the use of HDF5. Unfortunately, HDF5 is not convenient to handle
structures such as arrays of variable-size elements, for instance,
array of strings.
Therefore, we decided to rely on our own binary format.
This document describes the binary formats in BEAT and the API required by BEAT to handle multiple backend implementations. The
package `beat.env.python27`_ provides the *reference* Python backend
implementation based on `Python 2.7`_.
Binary Format
-------------
Our binary format does *not* contains information about the format of the data
itself, and it is hence necessary to know this format a priori. This means that
the format cannot be inferred from the content of a file.
We rely on the following fundamental C-style formats:
* int8
* int16
* int32
* int64
* uint8
* uint16
* uint32
* uint64
* float32
* float64
* complex64 (first real value, and then imaginary value)
* complex128 (first real value, and then imaginary value)
* bool (written as a byte)
* string
An element of such a basic format is written in the C-style way, using
little-endian byte ordering.
Besides, dataformats always consist of arrays or dictionary of such fundamental
formats or compound formats.
An array of elements is saved as followed. First, the shape of the array is
saved using an *uint64* value for each dimension. Next, the elements of the
arrays are saved in C-style order.
A dictionary of elements is saved as followed. First, the key are ordered
according to the lexicographic ordering. Then, the values associated to each of
these keys are saved following this ordering.
The platform is data-driven and always processes chunks of data. Therefore,
data are always written by chunks, each chunk being preceded by a text-formated
header indicated the start- and end- indices followed by the size (in bytes) of
the chunck.
Considering the Python backend of the platform, this binary format has been
successfully implemented using the ``struct`` module.
Filesystem Organization
-----------------------
......
......@@ -30,7 +30,7 @@ Go through the following sequence of commands:
.. code-block:: sh
$ git co https://gitlab.idiap.ch/bob/bob.admin
$ git checkout https://gitlab.idiap.ch/bob/bob.admin
$ #install miniconda (version 4.4 or above required)
$ conda activate base
$ cd beat.backend.python #cd into this package's sources
......@@ -73,7 +73,7 @@ base images before you try to run tests or experiments on your computer::
$ docker pull docker.idiap.ch/beat/beat.env.cxx:1.0.2
Optionally, also download the following images to be able to re-run experiments
downloaded from the BEAT platform (not required for unit testing)::
downloaded from the BEAT platform (not required for unit testing). These docker images corresponds to the python environment available on the platform. Keep in mind that at the moment you cannot use different environment to run each block when you are using BEAT locally::
$ docker pull docker.idiap.ch/beat/beat.env.python:0.0.4
$ docker pull docker.idiap.ch/beat/beat.env.python:1.0.0
......@@ -90,6 +90,7 @@ To build the documentation, just do:
$ ./bin/sphinx-build doc sphinx
Testing
-------
......@@ -103,18 +104,18 @@ use ``nose``:
.. note::
Some of the tests for our command-line toolkit require a running BEAT
platform web-server, with a compatible ``beat.core`` installed (preferably
the same). By default, these tests will be skipped. If you want to run
them, you must setup a development web server and set the environment
variable ``BEAT_CORE_TEST_PLATFORM`` to point to that address. For example::
Some of the tests for our command-line toolkit require a running BEAT
platform web-server, with a compatible ``beat.core`` installed (preferably
the same). By default, these tests will be skipped. If you want to run
them, you must setup a development web server and set the environment
variable ``BEAT_CORE_TEST_PLATFORM`` to point to that address. For example::
$ export BEAT_CORE_TEST_PLATFORM="http://example.com/platform/"
$ ./bin/nosetests -sv
$ export BEAT_CORE_TEST_PLATFORM="http://example.com/platform/"
$ ./bin/nosetests -sv
.. warning::
.. warning::
Do **NOT** run tests against a production web server.
Do **NOT** run tests against a production web server.
If you want to skip slow tests (at least those pulling stuff from our servers)
......@@ -131,15 +132,13 @@ To measure the test coverage, do the following::
Our documentation is also interspersed with test units. You can run them using
sphinx::
$ ./bin/sphinx -b doctest doc sphinx
Other bits
----------
$ ./bin/sphinx -b doctest doc sphinx
Other Bits
==========
Profiling
==========
---------
In order to profile the test code, try the following::
......
......@@ -33,7 +33,6 @@ This package provides the core components of BEAT ecosystem. These core componen
.. toctree::
introduction
io
backend_api
develop
api
......
......@@ -26,10 +26,10 @@
==============
Introduction
==============
A typical BEAT experiment is composed of several building blocks. Datasets that provide data to the system, algorithms that handles the functions introduced by user, analyzers that is in charge of interpreting the output result and producing the appropriate metrics and figures, and toolchains that determines the data flow between the blocks from datasets to the final results. In addition, each block accepts specific data formats and the data is synchronized between blocks neatly without users need to interfere. These basic functionalities that are introduced in :ref:`beat-system` are all defined and managed by ``beat.core``. For example, as it is explained in :ref:`beat-system-algorithms`, algorithm objects should be derived from the class
A typical BEAT experiment is composed of several building blocks. Datasets that provide data to the system, algorithms that handles the functions introduced by user, analyzers that is in charge of interpreting the output result and producing the appropriate results and figures, and toolchains that determines the data flow between the blocks from datasets to the final results. In addition, each block accepts specific data formats and the data is synchronized between blocks neatly without users need to interfere. These basic functionalities that are introduced in :ref:`beat-system` are all defined and managed by ``beat.core``. For example, as it is explained in :ref:`beat-system-algorithms`, algorithm objects should be derived from the class
``Algorithm`` when using Python or in case of C++, they should be derived from ``IAlgorithmLagacy``, ``IAlgorithmSequential``, or ``IAlgorithmAutonomous`` depending of the algorithm type. All these parent classes are defined in ``beat.core`` package.
The rest of this document includes information for developers for local development of the package and its API.
The rest of this document includes information about the backend api used to handle data through the BEAT ecosystem. For developers and advanced user there is information for local development of the package.
.. include:: links.rst
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment