Commit 0edcbd49 authored by André Anjos's avatar André Anjos 💬
Browse files

Merge branch 'docs' into 'master'

merge new documentation to master

See merge request !265
parents b0b627f2 c66e0439
Pipeline #25511 passed with stages
in 15 minutes and 25 seconds
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
###############################################################################
# #
# Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ #
# Contact: beat.support@idiap.ch #
# #
# This file is part of the beat.web module of the BEAT platform. #
# #
# Commercial License Usage #
# Licensees holding valid commercial BEAT licenses may use this file in #
# accordance with the terms contained in a written agreement between you #
# and Idiap. For further information contact tto@idiap.ch #
# #
# Alternatively, this file may be used under the terms of the GNU Affero #
# Public License version 3 as published by the Free Software and appearing #
# in the file LICENSE.AGPL included in the packaging of this file. #
# The BEAT platform is distributed in the hope that it will be useful, but #
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY #
# or FITNESS FOR A PARTICULAR PURPOSE. #
# #
# You should have received a copy of the GNU Affero Public License along #
# with the BEAT platform. If not, see http://www.gnu.org/licenses/. #
# #
###############################################################################
import os
import sys
import pkg_resources
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '1.3'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.ifconfig',
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.napoleon',
'sphinx.ext.viewcode',
'sphinxcontrib.programoutput',
]
import sphinx
if sphinx.__version__ >= "1.4.1":
extensions.append('sphinx.ext.imgmath')
else:
extensions.append('sphinx.ext.pngmath')
# Always includes todos
todo_include_todos = True
# Create numbers on figures with captions
numfig = True
# If we are on OSX, the 'dvipng' path maybe different
dvipng_osx = '/opt/local/libexec/texlive/binaries/dvipng'
if os.path.exists(dvipng_osx): pngmath_dvipng = dvipng_osx
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
top_module = 'beat'
# General information about the project.
project = u'BEAT Web Application' # (%s)' % top_module
authors = u'Idiap Research Institute'
import time
copyright = u'%s, Idiap Research Institute, Switzerland' % time.strftime('%Y')
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# Grab the setup entry
distribution = pkg_resources.require('beat.web')[0]
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = distribution.version
# The full version, including alpha/beta/rc tags.
release = distribution.version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Autodoc settings ---------------------------------------------------
autodoc_member_order = 'bysource'
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
import sphinx_rtd_theme
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'BEATdoc'
# -- Options for LaTeX output --------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
'papersize': 'a4paper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
'preamble': '\setcounter{tocdepth}{4}',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
(master_doc, top_module + '.tex', project, authors, 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, top_module, project, authors, 1),
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, top_module, project, authors, top_module, project, 'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
def smaller_than(v1, v2):
"""Compares scipy/numpy version numbers"""
c1 = v1.split('.')
c2 = v2.split('.')[:len(c1)] #clip to the compared version
for i, k in enumerate(c2):
n1 = c1[i]
n2 = c2[i]
try:
n1 = int(n1)
n2 = int(n2)
except ValueError:
n1 = str(n1)
n2 = str(n2)
if n1 > n2: return False
return True
# Some name mangling to find the correct sphinx manuals for some packages
numpy_version = __import__('numpy').version.version
if smaller_than(numpy_version, '1.5.z'):
numpy_version = '.'.join(numpy_version.split('.')[:-1]) + '.x'
else:
numpy_version = '.'.join(numpy_version.split('.')[:-1]) + '.0'
numpy_manual = 'http://docs.scipy.org/doc/numpy-%s/' % numpy_version
# For inter-documentation mapping:
intersphinx_mapping = {
'http://docs.python.org/%d.%d/' % sys.version_info[:2]: None,
numpy_manual: None,
'http://matplotlib.sourceforge.net/': None,
}
.. vim: set fileencoding=utf-8 :
.. _beat_web:
======================
BEAT Web Application
======================
This documentation includes information about the BEAT platform.
For users
=========
.. toctree::
:maxdepth: 1
:titlesonly:
user/index.rst
For developers
==============
.. toctree::
:maxdepth: 1
:titlesonly:
admin/index.rst
api/index.rst
This diff is collapsed.
......@@ -36,7 +36,7 @@ configuration page, the declaration of this experiment transmitted to the
scheduler, that now must run the experiment until it finishes, you press the
``stop`` button, or an error condition is produced.
As it is described in the :ref:`toolchains` section, the scheduler first breaks
As it is described in the :ref:`beat-system-toolchains` section, the scheduler first breaks
the toolchain into a sequence of executable blocks with dependencies. For
example: block ``B`` must be run after block ``A``. Each block is then
scheduled for execution depending on current resource availability. If no more
......
......@@ -30,37 +30,15 @@
Data formats specify the transmitted data between the blocks of a toolchain.
They describe the format of the data blocks that circulate between algorithms
and formalize the interaction between algorithms and data sets, so they can
communicate in an orderly manner. Inputs and outputs of the algorithms and
datasets **must** be formally declared. Two algorithms that communicate
directly must produce and consume the **same** type of data objects.
communicate in an orderly manner. For more detailed information see :ref:`beataformats`.
A data format specifies a list of typed fields. An algorithm or data set
generating a block of data (via one of its outputs) must fill all the fields
declared in that data format. An algorithm consuming a block of data (via one
of its inputs) must not expect the presence of any other field than the ones
defined by the data format.
The |project| platform provides a number of pre-defined formats to facilitate
experiments. They are implemented in an extensible way. This allows users to
define their own formats, based on existing ones, while keeping some level of
compatibility with other existing algorithms.
.. note:: **Naming Convention**
.. note::
Data formats are named using three values joined by a ``/`` (slash)
operator:
* **username**: indicates the author of the dataformat
* **name**: an identifier for the object
* **version**: an integer (starting from one), indicating the version of
the object
operator. The first value is the **username**.
Each tuple of these three components defines a *unique* data format name
inside the platform. For example, ``system/float/1``.
The ``system`` user, provides a number of `pre-defined formats such as
integers, booleans, floats and arrays
<https://www.beat-eu.org/platform/dataformats/system/>`_. You may also
The ``system`` user, provides a number of pre-defined formats such as
integers, booleans, floats and arrays (see `here <https://www.beat-eu.org/platform/dataformats/system/>`_). You may also
browse `publicly available data formats`_ to see all available data formats
from the ``system`` and other users.
......@@ -74,218 +52,6 @@ to that data format, like shown on the image below:
.. image:: img/system-defined-info.*
A data format is declared as a JSON_ object with several fields. For example,
the following declaration could represent the coordinates of a bounding box in
a video frame:
.. code-block:: javascript
{
"value": [
0,
{
"frame_id": "uint64",
"height": "int32",
"width": "int32",
"top-left-y": "int32",
"top-left-x": "int32"
}
]
}
The special field ``#description`` can be used to store a short description of
the declared data format. It is ignored in practice and only used for
informational purposes. Each field in a declaration has a well-defined type, as
explained next.
Simple type (primitive object)
==============================
The |project| platform supports the following *primitive* types.
* signed integers: ``int8``, ``int16``, ``int32``, ``int64``
* unsigned integers: ``uint8``, ``uint16``, ``uint32``, ``uint64``
* floating-point numbers: ``float32``, ``float64``
* complex numbers: ``cpmplex64``, ``complex128``
* a boolean value: ``bool``
* a string: ``string``
Aggregation
===========
A data format can be composed of complex objects formed by aggregating other
*declared* types. For example, we could define the positions of the eyes of a
face in an image like this:
.. code-block:: javascript
{
"left": "system/coordinates/1",
"right": "system/coordinates/1"
}
Arrays
======
A field can be a multi-dimensional array of any other type. Here ``array1`` is
declared as a one dimensional array of 10 32-bit signed integers (``int32``)
and ``array2`` as a two-dimensional array with 10 rows and 5 columns of
booleans:
.. code-block:: javascript
{
"array1": [10, "int32"],
"array2": [10, 5, "bool"]
}
An array can have up to 32 dimensions. It can also contain objects (either
declared inline, or using another data format). It is also possible to declare
an array without specifying the number of elements in some of its dimensions,
by using a size of 0 (zero). For example, here is a two-dimensional grayscale
image of unspecified size:
.. code-block:: javascript
{
"value": [0, 0, "uint8"]
}
You may also fix the some of dimensions extent. For example, here is a
possible representation for a three-dimensional RGB image of unspecified size
(width and height):
.. code-block:: javascript
{
"value": [3, 0, 0, "uint8"],
}
In this representation, the image must have 3 color planes (no more, no less).
The width and the height are unspecified.
.. note:: **Unspecified Dimensions**
Because of the way the |project| platform stores data, not all combinations
of unspecified extents will work for arrays. As a rule of thumb, only the
last dimensions may remain unspecified. These are valid:
.. code-block:: javascript
{
"value1": [0, "float64"],
"value2": [3, 0, "float64"],
"value3": [3, 2, 0, "float64"],
"value4": [3, 0, 0, "float64"],
"value5": [0, 0, 0, "float64"]
}
Whereas this would be invalid declarations for arrays:
.. code-block:: javascript
{
"value": [0, 3, "float64"],
"value": [4, 0, 3, "float64"]
}
Object Representation
---------------------
As you'll read in our :ref:`Algorithms` section, data is available via our
backend API to the user algorithms. For example, in Python, the |project|
platform uses NumPy_ data types to pass data to and from algorithms. For
example, when the algorithm reads data for which the format is defined like:
.. code-block:: javascript
{
"value": "float64"
}
The field ``value`` of an instance named ``object`` of this format is
accessible as ``object.value`` and will have the type ``numpy.float64``. If the
format would be, instead:
.. code-block:: javascript
{
"value": [0, 0, "float64"]
}
It would be accessed in the same way (i.e., via ``object.value``), except that
the type would be ``numpy.ndarray`` and ``object.value.dtype`` would be
``numpy.float64``. Naturally, objects which are instances of a format like
this:
.. code-block:: javascript
{
"x": "int32",
"y": "int32"
}
Could be accessed like ``object.x``, for the ``x`` value and ``object.y``, for
the ``y`` value. The type of ``object.x`` and ``object.y`` would be
``numpy.int32``.
Conversely, if you *write* output data in an algorithm, the type of the output
objects are checked for compatibility with respect to the value declared on the
format. For example, this would be a valid use of the format above, in Python:
.. code-block:: python
import numpy
class Algorithm:
def process(self, inputs, outputs):
# read data
# prepares object to be written
myobj = {"x": numpy.int32(4), "y": numpy.int32(6)}
# write it
outputs["point"].write(myobj) #OK!
If you try to write into an object that is supposed to be of type ``int32``, a
``float64`` object, an exception will be raised. For example:
.. code-block:: python
import numpy
class Algorithm:
def process(self, inputs, outputs):
# read data
# prepares object to be written
myobj = {"x": numpy.int32(4), "y": numpy.float64(3.14)}
# write it
outputs["point"].write(myobj) #Error: cannot downcast!
The bottomline is: **all type casting in the platform must be explicit**. It
will not automatically downcast or upcast objects for you as to avoid
unexpected precision loss leading to errors.
Editing Operations
......
......@@ -37,23 +37,6 @@ such as different databases and algorithms. Each experiment has its own
:ref:`toolchains` which cannot be changed after the experiment is created.
Experiments can be shared and forked, to ensure maximum re-usability.
.. note:: **Naming Convention**
Experiments are named using five values joined by a ``/`` (slash)
operator:
* **username**: indicates the author of the experiment
* **toolchain username**: indicates the author of the toolchain used for
that experiment
* **toolchain name**: indicates the name of the toolchain used for that
experiment
* **toolchain version**: indicates the version (integer starting from