Commit 732c6411 authored by André Anjos's avatar André Anjos 💬 Committed by Samuel GAIST

Doc revamp and fixing

parent a2bcf918
......@@ -25,5 +25,6 @@
# #
###############################################################################
#see http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
__import__('pkg_resources').declare_namespace(__name__)
# see https://docs.python.org/3/library/pkgutil.html
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
......@@ -24,7 +24,3 @@
# with the BEAT platform. If not, see http://www.gnu.org/licenses/. #
# #
###############################################################################
#see http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
__import__('pkg_resources').declare_namespace(__name__)
......@@ -111,11 +111,11 @@ def pull(webapi, prefix, names, force, indentation, format_cache, lib_cache):
prefix (str): A string representing the root of the path in which the user
objects are stored
names (list): A list of strings, each representing the unique relative path
of the objects to retrieve or a list of usernames from which to retrieve
objects. If the list is empty, then we pull all available objects of a
given type. If no user is set, then pull all public objects of a given
type.
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
force (bool): If set to ``True``, then overwrites local changes with the
remotely retrieved copies.
......
......@@ -233,13 +233,14 @@ def retrieve_remote_list(webapi, type, fields):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
fields (list of str): A list of fields to retrieve from the remote server
fields (:py:class:`list`): A list of fields to retrieve from the remote
server
Returns:
list: A list of dictionaries containing the ``name``, ``short_description``
and ``hash`` of available remote objects.
:py:class:`list`: A list of dictionaries containing the ``name``,
``short_description`` and ``hash`` of available remote objects.
'''
......@@ -279,14 +280,14 @@ def make_up_remote_list(webapi, type, requirements):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
requirements (list of str): A list of requirements that are used to filter
(additively) the available (remote) objects.
requirements (:py:class:`list`): A list of requirements that are used to
filter (additively) the available (remote) objects.
Returns:
list: A list of valid object names matching user requirements and its
order.
:py:class:`list`: A list of valid object names matching user requirements
and its order.
'''
......@@ -369,14 +370,15 @@ def make_up_local_list(prefix, type, requirements):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
requirements (list of str): A list of requirements that are used to filter
(additively) the available (remote) objects.
requirements (:py:class:`list`): A list of requirements that are used to
filter (additively) the available (remote) objects.
Returns:
list: A list of strings, each with the relative name of an object belonging
to a certain category and in the order prescribed by the user.
:py:class:`list`: A list of strings, each with the relative name of an
object belonging to a certain category and in the order prescribed by the
user.
'''
......@@ -568,7 +570,7 @@ def check(prefix, type, names):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to check. If the list is empty, then we
check all available objects of a given type.
......@@ -598,7 +600,8 @@ def fetch_object(webapi, type, name, fields):
name (str): A string defining the name of the object to retrieve
fields (list of str): A list of fields to retrieve from the remote server
fields (:py:class:`list`): A list of fields to retrieve from the remote
server
Returns:
......@@ -637,13 +640,13 @@ def pull(webapi, prefix, type, names, fields, force, indentation):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
fields (list of str): A list of strings, each defining one field that
fields (:py:class:`list`): A list of strings, each defining one field that
**must** be downloaded from the web-server for a given object of the
current type and passed, unchanged to the storage ``save()`` method. For
example, for toolchains, this value shall be ``['declaration']``. For
......@@ -663,10 +666,10 @@ def pull(webapi, prefix, type, names, fields, force, indentation):
calling process. This value should be zero if everything works OK,
otherwise, different than zero (POSIX compliance).
list: A list of strings containing the names of objects successfuly
downloaded or which were already present on the current installation (if
the user has chosen not to ``--force`` the override), in the order of
their download.
:py:class:`list`: A list of strings containing the names of objects
successfuly downloaded or which were already present on the current
installation (if the user has chosen not to ``--force`` the override), in
the order of their download.
"""
......@@ -714,7 +717,7 @@ def diff(webapi, prefix, type_, name, fields):
name (str): A string defining the name of the object to calculate
differences from.
fields (list of str): A list of strings, each defining one field that
fields (:py:class:`list`): A list of strings, each defining one field that
**must** be downloaded from the web-server for a given object of the
current type. For example, for toolchains, this value shall be
``['declaration']``. For algorithms, it shall be ``['declaration',
......@@ -954,7 +957,7 @@ def delete_local(prefix, type, names):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to delete.
......@@ -995,7 +998,7 @@ def delete_remote(webapi, type, names):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to delete.
......@@ -1052,8 +1055,8 @@ def status(webapi, prefix, type):
calling process. This value should be zero if everything works OK,
otherwise, different than zero (POSIX compliance).
list: A list of objects that have local modifications and should be pushed
remotely, eventually.
:py:class:`list`: A list of objects that have local modifications and
should be pushed remotely, eventually.
"""
......@@ -1123,13 +1126,13 @@ def push(webapi, prefix, type, names, fields, mappings, force, dry_run,
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to push or a filtering criteria for local
objects. If the list is empty, then we push all available objects of a
given type, which have changes. If no user is set, then an error is
raised.
fields (list of str): A list of strings, each defining one field that
fields (:py:class:`list`): A list of strings, each defining one field that
**must** be uploaded to the web-server for a given object of the current
type. For example, for toolchains, this value shall be ``['declaration',
'description']``. For algorithms, it shall be ``['declaration', 'code',
......@@ -1156,9 +1159,9 @@ def push(webapi, prefix, type, names, fields, mappings, force, dry_run,
calling process. This value should be zero if everything works OK,
otherwise, different than zero (POSIX compliance).
list: A list of strings containing the names of objects successfuly
uploaded or which were already present on the remote server (if
the user has chosen not to ``--force`` the override), in the order of
:py:class:`list`: A list of strings containing the names of objects
successfuly uploaded or which were already present on the remote server
(if the user has chosen not to ``--force`` the override), in the order of
their upload.
"""
......@@ -1283,7 +1286,7 @@ def dot_diagram(prefix, type, names, path, formats):
type (str): One of ``database``, ``dataformat``, ``algorithm``,
``toolchain`` or ``experiment``.
names (list of str): A list of strings, each representing the unique
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to push or a filtering criteria for local
objects. If the list is empty, then we push all available objects of a
given type, which have changes. If no user is set, then an error is
......@@ -1293,8 +1296,8 @@ def dot_diagram(prefix, type, names, path, formats):
assigned correspond to the full object name. If not set, the default is
to write on the current directory.
formats (list of str): A list of formats to dump. If not set or set to an
empty value, then dump dot/graphviz and png formats.
formats (:py:class:`list`): A list of formats to dump. If not set or set to
an empty value, then dump dot/graphviz and png formats.
Returns:
......@@ -1356,8 +1359,8 @@ def stringify(value):
Parameters:
value (dict): A dictionary representing the baseformat object, obtained as
with :py:meth:`beat.core.baseformat.baseformat.as_dict`, that represents
the object one seeks to represent in string format.
with :py:meth:`beat.backend.python.baseformat.baseformat.as_dict`, that
represents the object one seeks to represent in string format.
Returns
......
......@@ -340,11 +340,11 @@ def pull(webapi, prefix, names, force, indentation, format_cache):
prefix (str): A string representing the root of the path in which the
user objects are stored
names (list): A list of strings, each representing the unique relative
path of the objects to retrieve or a list of usernames from which to
retrieve objects. If the list is empty, then we pull all available
objects of a given type. If no user is set, then pull all public
objects of a given type.
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
force (bool): If set to ``True``, then overwrites local changes with the
remotely retrieved copies.
......
......@@ -87,11 +87,11 @@ def pull(webapi, prefix, names, force, indentation, cache):
prefix (str): A string representing the root of the path in which the user
objects are stored
names (list): A list of strings, each representing the unique relative path
of the objects to retrieve or a list of usernames from which to retrieve
objects. If the list is empty, then we pull all available objects of a
given type. If no user is set, then pull all public objects of a given
type.
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
force (bool): If set to ``True``, then overwrites local changes with the
remotely retrieved copies.
......
......@@ -363,11 +363,11 @@ def pull(webapi, prefix, names, force, indentation, format_cache):
prefix (str): A string representing the root of the path in which the
user objects are stored
names (list): A list of strings, each representing the unique relative
path of the objects to retrieve or a list of usernames from which to
retrieve objects. If the list is empty, then we pull all available
objects of a given type. If no user is set, then pull all public
objects of a given type.
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
force (bool): If set to ``True``, then overwrites local changes with the
remotely retrieved copies.
......
......@@ -82,11 +82,11 @@ def pull(webapi, prefix, names, force, indentation, cache):
prefix (str): A string representing the root of the path in which the user
objects are stored
names (list): A list of strings, each representing the unique relative path
of the objects to retrieve or a list of usernames from which to retrieve
objects. If the list is empty, then we pull all available objects of a
given type. If no user is set, then pull all public objects of a given
type.
names (:py:class:`list`): A list of strings, each representing the unique
relative path of the objects to retrieve or a list of usernames from
which to retrieve objects. If the list is empty, then we pull all
available objects of a given type. If no user is set, then pull all
public objects of a given type.
force (bool): If set to ``True``, then overwrites local changes with the
remotely retrieved copies.
......
......@@ -26,13 +26,10 @@ Algorithms
The commands available for algorithms are:
.. command-output:: ./bin/beat algorithms --help
:cwd: ..
.. command-output:: beat algorithms --help
For instance, a list of the algorithms available on the remote platform can
be obtained as follows:
.. command-output:: ./bin/beat algorithms list --remote
:cwd: ..
.. command-output:: beat algorithms list --remote
.. vim: set fileencoding=utf-8 :
.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ ..
.. Contact: beat.support@idiap.ch ..
.. ..
.. This file is part of the beat.cmdline module of the BEAT platform. ..
.. ..
.. Commercial License Usage ..
.. Licensees holding valid commercial BEAT licenses may use this file in ..
.. accordance with the terms contained in a written agreement between you ..
.. and Idiap. For further information contact tto@idiap.ch ..
.. ..
.. Alternatively, this file may be used under the terms of the GNU Affero ..
.. Public License version 3 as published by the Free Software and appearing ..
.. in the file LICENSE.AGPL included in the packaging of this file. ..
.. The BEAT platform is distributed in the hope that it will be useful, but ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE. ..
.. ..
.. You should have received a copy of the GNU Affero Public License along ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/. ..
API
---
This section includes information for using the Python API of ``beat.cmdline``.
.. notice order is important!
.. automodule:: beat.cmdline.algorithms
.. automodule:: beat.cmdline.cache
.. automodule:: beat.cmdline.common
.. automodule:: beat.cmdline.config
.. automodule:: beat.cmdline.databases
.. automodule:: beat.cmdline.dataformats
.. automodule:: beat.cmdline.experiments
.. automodule:: beat.cmdline.libraries
.. automodule:: beat.cmdline.plotters
.. automodule:: beat.cmdline.status
.. automodule:: beat.cmdline.toolchains
.. automodule:: beat.cmdline.version
.. automodule:: beat.cmdline.webapi
......@@ -54,6 +54,11 @@ extensions = [
#'matplotlib.sphinxext.plot_directive',
]
# Use buildout-installed version of beat cmdline if available
bindir = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'bin')
if os.path.exists(bindir):
os.environ['PATH'] = bindir + os.pathsep + os.environ['PATH']
# Be picky about warnings
nitpicky = True
......@@ -294,5 +299,30 @@ def member_function_test(app, what, name, obj, skip, options):
return skip
return False
# Excludes the "Usage" docstrings from these modules to appear in the API
# documentation
exclude_docstrings = {
'beat.cmdline.databases': 'module',
'beat.cmdline.databases.process': 'function',
'beat.cmdline.algorithms': 'module',
'beat.cmdline.algorithms.process': 'function',
'beat.cmdline.libraries': 'module',
'beat.cmdline.libraries.process': 'function',
'beat.cmdline.toolchains': 'module',
'beat.cmdline.toolchains.process': 'function',
'beat.cmdline.experiments': 'module',
'beat.cmdline.experiments.process': 'function',
'beat.cmdline.plotters': 'module',
'beat.cmdline.plotters.process': 'function',
'beat.cmdline.config': 'module',
'beat.cmdline.config.process': 'function',
}
def remove_module_docstring(app, what, name, obj, options, lines):
if name in exclude_docstrings and what == exclude_docstrings[name]:
del lines[:]
def setup(app):
app.connect('autodoc-skip-member', member_function_test)
app.connect('autodoc-process-docstring', remove_module_docstring)
......@@ -29,9 +29,8 @@ Configuration
The ``beat`` command-line utility can operate independently of any initial
configuration. For example:
.. command-output:: ./bin/beat df list --remote
.. command-output:: beat df list --remote
:ellipsis: 10
:cwd: ..
By default, the ``beat`` application is pre-configured to access the `main BEAT
......@@ -58,8 +57,7 @@ that directory, do the following from your shell:
You can verify your username and token have been memorized on that working
directory with:
.. command-output:: ./bin/beat config list
:cwd: ..
.. command-output:: beat config list
By default, the command-line program considers your current working directory
......@@ -93,10 +91,7 @@ Assuming your username is "user" and you extracted the database files to
You may explore different configuration options with the ``--help`` flag of
``beat config``:
.. command-output:: ./bin/beat config --help
:cwd: ..
.. command-output:: beat config --help
.. Place your links here
.. _main beat website: https://wwww.beat-eu.org/platform
.. include:: links.rst
......@@ -26,14 +26,12 @@ Databases
The commands available for databases are:
.. command-output:: ./bin/beat databases --help
:cwd: ..
.. command-output:: beat databases --help
For instance, a list of the databases available on the remote platform can
be obtained as follows:
.. command-output:: ./bin/beat databases list --remote
:cwd: ..
.. command-output:: beat databases list --remote
Creating a new database
......@@ -45,7 +43,7 @@ Once done, issue the following sequence of commands to upload it to the server:
.. code-block:: sh
$ ./bin/beat -p prefix -m <platform> -t <your-token> databases push <db>/1
$ beat -p prefix -m <platform> -t <your-token> databases push <db>/1
Replace the string ``<db>`` with the fully qualified name of your database. For
......@@ -72,13 +70,13 @@ the new version.
.. code-block:: sh
$ ./bin/beat -p prefix -m <platform> -t <your-token> databases pull <db>/1
$ beat -p prefix -m <platform> -t <your-token> databases pull <db>/1
...
$ ./bin/beat -p prefix databases version <db>
$ beat -p prefix databases version <db>
...
$ vim prefix/databases/<db>/2.*
# once you're happy, upload the new version
$ ./bin/beat -p prefix -m <platform> -t <your-token> databases push <db>/2
$ beat -p prefix -m <platform> -t <your-token> databases push <db>/2
Replace the string ``<db>`` with the name of your database. For example,
......
......@@ -26,13 +26,13 @@ Dataformats
The commands available for data formats are:
.. command-output:: ./bin/beat dataformats --help
.. command-output:: beat dataformats --help
:cwd: ..
For instance, a list of the data formats available on the remote platform can
be obtained as follows:
.. command-output:: ./bin/beat dataformats list --remote
.. command-output:: beat dataformats list --remote
:cwd: ..
......@@ -38,8 +38,7 @@ experiments.
The commands available for experiments are:
.. command-output:: ./bin/beat experiments --help
:cwd: ..
.. command-output:: beat experiments --help
.. _beat-cmdline-experiments-running:
......@@ -97,5 +96,5 @@ How to examine the content of a data file?
The ``beat cache`` collection of commands interact with the cache:
.. command-output:: ./bin/beat cache --help
:cwd: ..
.. command-output:: beat cache --help
......@@ -31,7 +31,7 @@ This package provides a Python-based client for BEAT's web service or locally
installed repositories. It allows users to list, validate, edit, download and
upload objects from remote BEAT instances, as well as running BEAT experiments
locally. It also doubles as a Python-client API for packages that need to
implement more advanced functionality than this client (`beat`) provides.
implement more advanced functionality than this client (``beat``) provides.
.. toctree::
......@@ -44,6 +44,8 @@ implement more advanced functionality than this client (`beat`) provides.
experiments
databases
walkthrough
api
Indices and tables
==================
......
......@@ -40,8 +40,10 @@ prefix is just a path to a known directory to which the user has write access,
so as to store objects which are downloaded from the web server. This is the
typical directory structure in a prefix directory:
.. program-output:: ls -l src/beat.core/beat/core/test/prefix/
:cwd: ..
.. program-output:: ls -l /Users/andre/conda/envs/beatcl-dev/lib/python*/site-packages/beat/core/test/prefix
:shell:
Each of the subdirectories in the prefix keeps only objects of a given type.
For example, the ``dataformats`` subdirectory keeps only data format objects,
......@@ -58,8 +60,7 @@ The ``beat`` command-line utility bridges user interaction with a remote BEAT
web platform and locally available objects in a seamless way. The program is
normally available in the Idiap work environment:
.. command-output:: ./bin/beat --help
:cwd: ..
.. command-output:: beat --help
The command-line interface is separated in subcommands, for acting on specific
objects in the platform. Actions can be driven to operate on locally installed
......@@ -67,5 +68,3 @@ or remotely available objects. You'll find detailed information about
sub-commands on specific sub-sections of this documentation dedicated to that
particular type of object. In this section, we cover basic usage and
configuration only.
......@@ -26,13 +26,10 @@ Libraries
The commands available for libraries are:
.. command-output:: ./bin/beat libraries --help
:cwd: ..
.. command-output:: beat libraries --help
For instance, a list of the libraries available on the remote platform can
be obtained as follows:
.. command-output:: ./bin/beat libraries list --remote
:cwd: ..
.. command-output:: beat libraries list --remote
.. _main beat website: https://wwww.beat-eu.org/platform
# Not available in Python 2.7, but ok in Python 3.x
py:exc TypeError
py:exc RuntimeError
py:exc ValueError
py:exc KeyError
py:class tuple
py:class list
......@@ -26,8 +26,7 @@ Toolchains
The commands available for toolchains are:
.. command-output:: ./bin/beat toolchains --help
:cwd: ..
.. command-output:: beat toolchains --help
.. _beat-cmdline-toolchains-checkscript:
......@@ -43,7 +42,7 @@ For example, we check a correct file (found in
.. code-block:: sh
$ ./bin/beat --prefix=src/beat.core/beat/core/test/ toolchains check \
$ beat --prefix=src/beat.core/beat/core/test/ toolchains check \
integers_addition
The toolchain is executable!
......@@ -56,19 +55,21 @@ Now we check a file that isn't a correctly formatted JSON file:
``src/beat.core/beat/core/test/toolchains/invalid/invalid.json``:
.. code-block:: json
{
"invalid": true,
}
{
"invalid": true,
}
.. code-block:: sh
$ ./bin/beat --prefix=src/beat.core/beat/core/test/ toolchains check \
invalid/invalid
The toolchain isn't valid, due to the following errors:
Failed to decode the JSON file 'beat/src/beat.core/beat/core/test/toolchains/invalid/invalid.json':
Expecting property name enclosed in double quotes: line 3 column 1 (char 23)
$ beat --prefix=src/beat.core/beat/core/test/ toolchains check invalid/invalid
The toolchain isn\'t valid, due to the following errors:
Failed to decode the JSON file \'beat/src/beat.core/beat/core/test/toolchains/invalid/invalid.json\':
Expecting property name enclosed in double quotes: line 3 column 1 (char 23)
Here we are told that something is wrong JSON-wise around the line 3, column 1
of the JSON file. The error is the ``,`` (comma) character: in JSON, the last
......@@ -113,11 +114,11 @@ explanations about the content of the declaration file):
]
}
.. code-block:: sh
$ ./bin/beat --prefix=src/beat.core/beat/core/test/ toolchains check \
invalid/empty_blocks_list
The toolchain isn't valid, due to the following errors:
$ beat --prefix=src/beat.core/beat/core/test/ toolchains check invalid/empty_blocks_list
The toolchain isn\'t valid, due to the following errors:
Unknown inputs: echo.in
Unknown result outputs: echo.out
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment