diff --git a/README.rst b/README.rst
index 7224cbfb32c7b1a59c542e5720a7a79ab721f192..9136dd3d9809a6e8a4eaf1ee5dc07fa66f7426e5 100644
--- a/README.rst
+++ b/README.rst
@@ -1,15 +1,17 @@
.. vim: set fileencoding=utf-8 :
-.. Thu 30 Jan 08:46:53 2014 CET
+.. Tue 27 Feb 2018 09:06:28 CET
-.. image:: http://img.shields.io/badge/docs-stable-yellow.png
+.. image:: https://img.shields.io/badge/docs-stable-yellow.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/stable/index.html
-.. image:: http://img.shields.io/badge/docs-latest-orange.png
+.. image:: https://img.shields.io/badge/docs-latest-orange.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/master/index.html
.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/build.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
+.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/coverage.svg
+ :target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
.. image:: https://img.shields.io/badge/gitlab-project-0000c0.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout
-.. image:: http://img.shields.io/pypi/v/bob.buildout.png
+.. image:: https://img.shields.io/pypi/v/bob.buildout.svg
:target: https://pypi.python.org/pypi/bob.buildout
@@ -18,157 +20,17 @@
===================================
This package is part of the signal-processing and machine learning toolbox
-Bob_. It contains a number of ``zc.buildout`` recipes for simplifying builds of
-Bob_ packages.
-
-
-C++/Python Extension
---------------------
-
-This extension allows you to compile C/C++ extensions that depend on each other
-using a buildout. It assures that eggs living in both ``develop-eggs`` and
-``eggs`` are on your path before building the packages in the ``develop``
-section. By using this extension you can drop the use of the local recipe
-``bob.buildout:develop``, which should be considered deprecated.
-
-
-Supported Options
-=================
-
-verbose
-
- If set, buildout it will output the compilation commands while compiling the
- module.
-
-prefixes
-
- A list of directories where this recipe will look for installed software,
- such as compiled libraries and header files. It is the same as setting the
- environment variable ``BOB_PREFIX_PATH`` to a list of paths containing
- externally installed software. As a side-effect, setting ``BOB_PREFIX_PATH``
- also sets, internally, ``PKG_CONFIG_PATH`` to a list of directories following
- where to search for pkg-config files.
-
-debug
-
- If set, the module will be compiled with debugging symbols and with
- optimization turned off. If ``debug`` is set to ``true``, this is equivalent
- to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
- -g``. If it is set to ``false``, it is the same as appending ``-O3 -g0``
- instead.
-
-environ
-
- The name of a section on your configuration file that contains the names and
- values of environment variables that should be used through the build. This
- section is named, by default, ``environ``.
-
- If a section named ``environ`` exists, it is read and the environment
- variables are set **before** the specified eggs are developed. You can use
- variable substitution on this section. Here is an an example::
-
- [environ]
- CFLAGS = '-O0 -g -DNDEBUG'
- CXXFLAGS = ${CFLAGS}
-
- Notice there is some functionality overlap between the previous flags and the
- use of section ``environ``. While it is more flexible, you must understand
- the consequences of setting both ``prefixes`` and ``debug``, together with
- ``environ``. The rule is simple: values set on the ``environ`` section have
- **precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
- ``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
- the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
- ``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
- ``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
- to those listed in ``prefixes``, if that is also set.
-
-
-Multi-Script Installer
-----------------------
-
-This recipe installs **all** most used scripts and interpreter proxies for your
-package. It will look at the ``buildout`` section entry called ``prefixes``,
-that potentially lists prefixes that should be **prepended** to the default
-python environment. In these prefixes, it will look for standard python
-directories. If one or more are found, these paths are **prepended** into
-the resulting scripts generated by this recipe and eggs will be searched on
-those locations prioritarily.
-
-By default, this recipe will use the eggs defined at the ``buildout`` section
-called ``eggs``, but that can be overriden locally. It generates these scripts:
-
-python
- A pre-configured python interpreter
-
-gdb-python
- A pre-configured python interpreter, prefixed with ``gdb`` to make debugging
- easier. Use it like you use ``python``.
-
-nosetests
- A test runner called ``nosetests`` will be created on the bin directory of
- buildout.
-
-coverage
- A test coverage application called ``coverage`` will be created on the bin
- directory of buildout.
-
-sphinx
- Several sphinx utilities will be created on the bin directory of buildout.
-
-package scripts
- Package scripts will be created taking into account the ``prefixes``
- established for this section or globally (as a second priority).
-
-To use this recipe, you just have to simply do::
-
- [scripts]
- recipe = bob.buildout:scripts
-
-
-Common Supported Options
-========================
-
-The recipe supports the following options:
-
-prefixes
- A list of directories where this recipe will look for subdirectories with
- the stem ``lib/python*/site-packages``. All directories matching this
- condition are appended to the search paths. If not given, the value of this
- property defaults to ``buildout.prefixes``. Both can be empty, which makes
- this recipe default to using standard available paths.
-
-eggs
- The eggs option specifies a list of eggs to use for **building** this
- package. Each string must be given on a separate line. If not given, the
- value of this property defaults to ``buildout.eggs``.
-
-dependent-scripts
- If set to the string ``true``, scripts will be generated for all required
- eggs in addition to the eggs specifically named.
-
-interpreter
- The name of a script to generate that allows access to a Python interpreter
- that has the path set based on the eggs installed. If you don't specify
- anything, the default value ``python`` will be used.
-
-extra-paths
- Extra paths to be appended in a generated script. To prepend, using the
- ``prefixes`` entry.
-
-nose-flags
- These are extra flags that are **appended** to the given ``nosetests``
- command line, automatically. Use this to preset arguments you like running
- all the time like ``-v``, for example.
+Bob_. It contains a number of ``zc.buildout`` recipes for simplifying
+development of Bob_ packages.
Installation
------------
-Follow our `installation`_ instructions. Then, using the Python interpreter
-provided by the distribution, bootstrap and buildout this package::
+Complete Bob's `installation`_ instructions. Then, to install this package,
+run::
- $ python bootstrap-buildout.py
- $ ./bin/buildout
+ $ conda install bob.buildout
Contact
@@ -180,5 +42,5 @@ development `mailing list`_.
.. Place your references here:
.. _bob: https://www.idiap.ch/software/bob
-.. _installation: https://gitlab.idiap.ch/bob/bob/wikis/Installation
-.. _mailing list: https://groups.google.com/forum/?fromgroups#!forum/bob-devel
+.. _installation: https://www.idiap.ch/software/bob/install
+.. _mailing list: https://www.idiap.ch/software/bob/discuss
diff --git a/doc/buildout.rst b/doc/buildout.rst
new file mode 100644
index 0000000000000000000000000000000000000000..3f23072ede33e926cb0032e64dad51bade2a8083
--- /dev/null
+++ b/doc/buildout.rst
@@ -0,0 +1,224 @@
+.. bob.buildout.recipes:
+
+
+=======================
+ Extension and Recipes
+=======================
+
+This package is composed of an extension to zc.buildout_ and a recipe. We
+recommend you use both during the creation of your own package-based recipes.
+Here is a typical ``buildout.cfg`` file that can be used to setup the
+development of your own package, and which we normally ship with all
+|project|-based packages:
+
+
+.. code-block:: ini
+
+ [buildout]
+ parts = scripts
+ develop = .
+ eggs = <PACKAGE>
+ extensions = bob.buildout
+ newest = false
+ verbose = true
+
+ [scripts]
+ recipe = bob.buildout:scripts
+
+
+Replace ``<PACKAGE>`` by your package name and you should be ready to run the
+``buildout`` application. If you're curious about zc.buildout_ and how to
+structure your recipe to take full advantage of it, we advise you seek
+documentation on that package on its website or by searching the internet.
+
+The above setup will include all required material for building both simple
+python or python/C/C++ packages from |project|.
+
+By using extension ``bob.buildout`` on the ``buildout`` section of your recipe,
+you ensure the current package will be built taking into consideration all
+required environment/compiler settings required by |project|. This extension is
+strictly required for packages containing C/C++ bindings, but it is harmless to
+include it in Python-only packages.
+
+The section ``scripts`` define a list of scripts that will be installed on the
+``bin`` directory. By default, we make sure all packages listed in
+``buildout.eggs`` are available, as well as typical packages required for
+development such as ``sphinx``, for documentation building, ``nose`` for
+running your test suite and ``coverage``, for code coverage reporting. You may
+augment the ``eggs`` entry on the ``buildout`` section if you'd like further
+packages to be installed on your development environment. It is your call. An
+example package that is often listed there is ``ipdb`` - the iPython based
+python debugger. The syntax of the ``eggs`` entry on the ``buildout`` section
+is one package per line.
+
+The entry ``develop`` on the ``buildout`` section indicates to buildout the
+root directories of packages that it should take, prioritarily, before
+considering potential versions installed on base python (conda) environment. It
+typically says just ``.`` (dot), as we're typically willing to *only* make
+buildout aware of our local checkout. It can include more directories (one per
+line), if we'd like to cross-develop more packages. That is what we do, for
+example, when using the extension ``mr.developer`` (see:
+:ref:`bob.extension` development). For each package listed in ``develop``,
+buildout will run the equivalent of ``cd <dir> && python setup.py develop`` on
+each package **before** trying to resolve the dependencies in ``eggs``. It will
+then consider those locally installed packages in final load-path scheme for
+your applications.
+
+In case you're curious, the ``buildout.newest`` flag is internal to
+zc.buildout_. It instructs the setuptools_ machinery to consider or not
+versions of packages currently installed. If ``newest=true``, then only the
+absolute newest packages in PyPI_ can satisfy the build. By default, we let
+this setting to ``false`` indicating that what is currently installed will be
+sufficient. Only change it if you know the implications. Note, for example, in
+this case, you may not be testing anymore, exclusively, against a
+|project|-agreed set of dependencies.
+
+The remainder flags and options are explained next.
+
+
+Supported extension options
+---------------------------
+
+Here is a list of supported options for the ``bob.buildout``. They should
+appear in the ``buildout`` section of your recipe. They are considered for as
+long as ``bob.buildout`` is listed as an extension.
+
+
+``verbose``
+
+ If set, buildout it will output the compilation commands while compiling the
+ module. This flag should go into the ``buildout`` section of your recipe.
+ Accepted values are ``true`` or ``false``.(the default).
+
+
+``prefixes``
+
+ A list of directories where this recipe will look for installed software,
+ such as compiled libraries and header files. It is important if you're
+ compiling a package that requires external software such as ffmpeg or blitz++
+ header files to be available. It is the same as setting the environment
+ variable ``BOB_PREFIX_PATH`` to a list of paths containing externally
+ installed software. As a side-effect, setting ``BOB_PREFIX_PATH`` also sets,
+ internally, ``PKG_CONFIG_PATH`` to a list of directories following where to
+ search for pkg-config files.
+
+ You typically don't need to set this manually since ``bob.extension`` is
+ smart enough to figure out where to find external libraries. This is even
+ more true if you're using conda-based installations as the bootstrapping
+ environment of your buildout.
+
+
+``debug``
+
+ If set, the module will be compiled with debugging symbols and with
+ optimization turned off. If ``debug`` is set to ``true``, this is equivalent
+ to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
+ -g``. By default, the value is ``false``. This setting is advised if you are
+ compiling python C/C++ bindings and would like to debug C/C++ code locally.
+
+
+``environ``
+
+ The name of another section on your configuration file that contains the
+ names and values of environment variables that should be set before any
+ possible build takes place. This section is named, by default, ``environ``.
+ That is, if you provide a section named ``environ`` on your buildout recipe,
+ then you don't need to explicitly specify its name using this flag.
+
+ If a section named ``environ`` (or whatever lists the ``environ`` override on
+ the ``buildout`` section of your recipe) exists, it is read and the
+ environment variables are set **before** the specified packages are built.
+ You can use variable substitution on this section. Here is an an example::
+
+ [environ]
+ CFLAGS = '-O0 -g -DNDEBUG'
+ CXXFLAGS = ${CFLAGS}
+
+ Notice there is some functionality overlap between the previous flags and the
+ use of section ``environ``. While it is more flexible, you must understand
+ the consequences of setting both ``prefixes`` and ``debug``, together with
+ ``environ``. The rule is simple: values set on the ``environ`` section have
+ **precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
+ ``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
+ the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
+ ``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
+ ``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
+ to those listed in ``prefixes``, if that is also set.
+
+
+The ``scripts`` recipe
+----------------------
+
+By using the recipe ``bob.buildout:scripts`` on one of the sections of your
+recipe, you ensure the scripts generated by the recipe will be built taking
+into consideration all installed packages from your base python environment
+(typically a conda-based installation). If you don't use the
+``bob.buildout:scripts`` recipe, zc.buildout_, by default, assumes no packages
+are availabe on the python installation and may download/recompile all
+dependencies from scratch.
+
+By default, this recipe will use the eggs defined at the ``buildout`` section
+called ``eggs``, but that can be overriden locally. It generates these scripts:
+
+``python``
+ A pre-configured python interpreter taking into consideration all eggs and
+ development packages on your recipe.
+
+``gdb-python`` or ``lldb-python``
+ A pre-configured python interpreter, prefixed with ``gdb`` (or ``lldb`` on a
+ MacOS system) to make debugging easier. Use it like you use ``python``.
+
+``nosetests``
+ A test runner called ``nosetests`` will be created on the bin directory of
+ buildout.
+
+``coverage``
+ A test coverage application called ``coverage`` will be created on the
+ ``bin`` directory of buildout.
+
+``sphinx-build`` and friends
+ Several sphinx utilities will be created on the bin directory of buildout.
+
+Other package scripts
+ Package scripts will be created taking into account the ``prefixes``
+ established for this section or globally (as a second priority).
+
+
+Supported recipe options
+========================
+
+The ``scripts`` recipe supports the following options (mostly for experts -
+read don't use them unless you know what you're doing):
+
+``prefixes``
+ Overrides or sets the list of prefixes in ``buildout.prefixes``. If not
+ given, the value of this property defaults to ``buildout.prefixes``. Both can
+ be empty, which makes this recipe default to using standard available paths.
+
+``eggs``
+ The eggs option specifies a list of eggs to use for **building** the scripts
+ of this recipe. Each string must be given on a separate line. If not given,
+ the value of this property defaults to ``buildout.eggs``.
+
+``dependent-scripts``
+ If set to the string ``true``, scripts will be generated for all required
+ eggs in addition to the eggs specifically named. By default, it is ``false``
+ to avoid the clutter it may cause on very high-level packages, with numerous
+ dependencies exporting scripts.
+
+``interpreter``
+ The name of a script to generate that allows access to a Python interpreter
+ that has the path set based on the eggs installed. If you don't specify
+ anything, the default value ``python`` will be used.
+
+``extra-paths``
+ Extra paths to be appended in a generated script. To prepend, use the
+ ``prefixes`` entry.
+
+``nose-flags``
+ These are extra flags that are **appended** to the given ``nosetests``
+ command line, automatically. Use this to preset arguments you like running
+ all the time like ``-v``, for example.
+
+
+.. include:: links.rst
diff --git a/doc/conf.py b/doc/conf.py
index 01172471d6f07d3b68b415634acc72e88ce58085..29d406d490c7f26d49d5c37da98be39138da2af1 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -4,6 +4,7 @@
import os
import sys
import glob
+import pkg_resources
# -- General configuration -----------------------------------------------------
@@ -27,6 +28,25 @@ extensions = [
'sphinx.ext.mathjax',
]
+# Be picky about warnings
+nitpicky = True
+
+# Ignores stuff we can't easily resolve on other project's sphinx manuals
+nitpick_ignore = []
+
+# Allows the user to override warnings from a separate file
+if os.path.exists('nitpick-exceptions.txt'):
+ for line in open('nitpick-exceptions.txt'):
+ if line.strip() == "" or line.startswith("#"):
+ continue
+ dtype, target = line.split(None, 1)
+ target = target.strip()
+ try: # python 2.x
+ target = unicode(target)
+ except NameError:
+ pass
+ nitpick_ignore.append((dtype, target))
+
# Always includes todos
todo_include_todos = True
@@ -57,14 +77,17 @@ project = u'bob.buildout'
import time
copyright = u'%s, Idiap Research Institute' % time.strftime('%Y')
+# Grab the setup entry
+distribution = pkg_resources.require(project)[0]
+
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
-version = open('../version.txt').read().strip()
+version = distribution.version
# The full version, including alpha/beta/rc tags.
-release = version
+release = distribution.version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
@@ -186,14 +209,49 @@ html_favicon = 'img/favicon.ico'
# Output file base name for HTML help builder.
htmlhelp_basename = project_variable + u'_doc'
+
# -- Post configuration --------------------------------------------------------
+# Included after all input documents
+rst_epilog = """
+.. |project| replace:: Bob
+.. |version| replace:: %s
+.. |current-year| date:: %%Y
+""" % (version,)
+
# Default processing flags for sphinx
autoclass_content = 'class'
autodoc_member_order = 'bysource'
autodoc_default_flags = [
'members',
'undoc-members',
- 'inherited-members',
'show-inheritance',
]
+
+# For inter-documentation mapping:
+intersphinx_mapping = {
+ 'python': ('https://docs.python.org/%d.%d/' % sys.version_info[:2], None),
+ 'six': ('https://six.readthedocs.io', None),
+ 'setuptools': ('https://setuptools.readthedocs.io/en/latest/', None),
+ 'bob.extension': ('http://www.idiap.ch/software/bob/docs/bob/bob.extension/stable/', None),
+ }
+
+# We want to remove all private (i.e. _. or __.__) members
+# that are not in the list of accepted functions
+accepted_private_functions = ['__array__']
+
+def member_function_test(app, what, name, obj, skip, options):
+ # test if we have a private function
+ if len(name) > 1 and name[0] == '_':
+ # test if this private function should be allowed
+ if name not in accepted_private_functions:
+ # omit privat functions that are not in the list of accepted private functions
+ return skip
+ else:
+ # test if the method is documented
+ if not hasattr(obj, '__doc__') or not obj.__doc__:
+ return skip
+ return False
+
+def setup(app):
+ app.connect('autodoc-skip-member', member_function_test)
diff --git a/doc/development.rst b/doc/development.rst
new file mode 100644
index 0000000000000000000000000000000000000000..c5c14d9378b0cd4aef687106e6a6663343753059
--- /dev/null
+++ b/doc/development.rst
@@ -0,0 +1,101 @@
+.. _bob.buildout.dev:
+
+=========================
+ Developing bob.buildout
+=========================
+
+You can quickly test this package by running the following commands:
+
+.. code-block:: sh
+
+ $ buildout
+ $ ./bin/nosetests -sv
+
+Testing is limited to certain internal functionality. If you want to do an
+extensive test and make sure changes are operating when you use this package to
+create an environment for *another* package, please read on.
+
+
+Cross-developing with another package
+-------------------------------------
+
+This is a chicken-and-egg problem as developing another package with a *new*
+version of this package requires a working installation of ``bob.buildout``. In
+order to break the loop, you'll need to first buildout with a simplified
+version of ``buildout.cfg``, to tell buildout ``bob.buildout`` is being
+developed alongside the test package. As an example, we'll develop
+``bob.buildout`` in a ``bob.extension`` checkout:
+
+.. code-block:: sh
+
+ $ git clone https://gitlab.idiap.ch/bob/bob.extension
+ $ cd bob.extension
+ # create the file first.cfg with the following contents:
+ $ cat first.cfg
+ [buildout]
+ parts =
+ develop = src/buildout
+ $ mkdir src
+ $ git clone https://gitlab.idiap.ch/bob/bob.buildout src/bob.buildout
+
+Setup your base conda-environment as usual and the, run ``buildout -c
+first.cfg``:
+
+
+.. code-block:: sh
+
+ $ buildout -c first.cfg
+
+
+The previous command should not download anything from PyPI_ and will create a
+symbolic egg link in ``develop-eggs`` called ``bob.buildout.egg-link``. To make
+sure your ``first.cfg`` bootstrap procedure worked, check there. Now, slightly
+modify ``buildout.cfg`` from ``bob.extension`` to include a new line in the
+``buildout.develop`` entry before ``.``, so the new buildout will also take the
+bootstrapped buildout into consideration. It should look like this:
+
+
+.. code-block:: sh
+
+ $ cat buildout.cfg
+ [buildout]
+ parts = scripts
+ develop = src/bob.buildout
+ .
+ eggs = bob.extension
+ extensions = bob.buildout
+ newest = false
+ verbose = true
+
+ [scripts]
+ recipe = bob.buildout:scripts
+ dependent-scripts = true
+
+
+Now run buildout normally, against the modified ``buildout.cfg``:
+
+.. code-block:: sh
+
+ $ buildout
+ ...
+
+
+This last step should provide you with a setup as performed by the
+bleeding-edge version of bob.buildout you have checked-out on the ``src``
+directory. You can modify it and re-run ``buildout`` until everything looks in
+order.
+
+
+Releasing bob.buildout
+----------------------
+
+This is a standard |project| package and therefore you **must** follow the
+standard releasing procedure using ``bob_new_version.py`` or any more recent
+script available for such purpose.
+
+You may create a ``bob.extension``-based environment for such as per
+instructions above or start from a pure conda-based install which contains
+``bob.extension`` and run that script from it.
+
+
+.. include:: links.rst
diff --git a/doc/guide.rst b/doc/guide.rst
new file mode 100644
index 0000000000000000000000000000000000000000..b21092a92fccba75ffe84e3784c6626a55924696
--- /dev/null
+++ b/doc/guide.rst
@@ -0,0 +1,218 @@
+.. bob.buildout.guide:
+
+=========================================
+ Local development of |project| packages
+=========================================
+
+Very often, developers of |project| packages are confronted with the need to
+clone repositories locally and develop installation/build and runtime code.
+While it is possible to use conda_ for such, the use of `zc.buildout`_ offers
+an quick and easy alternative to achieve this. It allows the creation of
+isolated, directory-based python development environments that can be modulated
+based on the development needs of the current package(s) one needs to work on.
+
+The steps involved in creating a development environment are the following:
+
+1. Checkout from gitlab_ the package the user wants to develop
+2. Create a conda installation containing base packages that the current
+ package being developed requires
+3. *Optionally*, create a buildout configuration that allows the
+ cross-development of packages
+4. Run the application ``buildout`` to set-up the desired development
+ environment
+
+This guide is a step-by-step guide in performing these.
+
+
+Checking out |project| package sources
+--------------------------------------
+
+|project| packages are developed through Gitlab_. In order to checkout a
+package, just use git_:
+
+
+.. code-block:: sh
+
+ $ git clone https://gitlab.idiap.ch/bob/<package>
+
+
+Where ``<package>`` is the package you want to develop. Various packages exist
+in |project|'s gitlab_ instance.
+
+
+Create a base conda-installation
+--------------------------------
+
+The base conda installation should contemplate all packages that you want to
+develop against. This is typically listed in the package's
+``requirements.txt``, but may also include test dependencies listed in
+``test-requirements.txt`` depending on the package. The file
+``conda/meta.yaml`` should be considered the canonical source for information
+concerning package installation and deployment. After following `Bob's
+installation`_ instructions, install all packages listed on the ``meta.yaml``
+file using a single conda installation command.
+
+For example, if the package one needs to develop lists build dependencies
+``A``, ``B`` and ``C`` and test dependencies ``T`` and ``U``, the following
+command-line should suffice once you followed `Bob's installation`_
+instructions:
+
+
+.. code-block:: sh
+
+ $ cd <package>
+ $ cat conda/meta.yaml #inspect contents and decide what to install
+ ...
+ $ conda create -n dev A B C T U bob.buildout
+ ...
+ $ source activate dev #ready to develop your package
+
+
+**Optional** Use of not-yet-released conda packages
+===================================================
+
+When developing and testing new features, one often wishes to work against the
+very latest, *bleeding edge*, available set of changes on dependent packages.
+If that is the case, consider adding our `conda beta channel`_ alongside conda
+channels available for download in your condarc_ file.
+
+The location of ``.condarc`` is configurable, but it is often set in
+``${HOME}/.condarc``. Read about condarc_ online if you are in doubt.
+
+After our `conda beta channel`_ is included on your configuration, proceed as
+above to create an environment with the latest dependencies instead of the
+latest *stable* versions of each package.
+
+
+**Optional** Automated environment creation
+===========================================
+
+It is possible to automate the above procedure using a script that tries to
+automatically parse build and test requirements in ``conda/meta.yaml`` and
+create the required development environment from scratch. This script lives in
+the package ``bob.admin``. The first step is to checkout ``bob.admin``
+alongside the package you're developing:
+
+
+.. code-block:: sh
+
+ $ cd ..
+ $ ls
+ <PACKAGE>
+ $ git checkout https://gitlab.idiap.ch/bob/bob.admin
+ $ ls
+ <PACKAGE>
+ bob.admin
+ $ cd <PACKAGE>
+
+
+.. note::
+
+ If you already have checked out ``bob.admin``, make sure to update it with
+ ``git pull``. We're constantly making improvements to this package.
+
+
+Once ``bob.admin`` is available alongside your package, make sure both
+``conda`` and ``conda-build`` are installed **and updated** on the root
+environment of your conda_ installation. The automated script requires conda_
+version 4.4 or above and ``conda-build`` version 3 or above. It also requires
+the package ``pyyaml`` to be installed on the root of your conda installation.
+Follow this recipe to get all up-to-date and ready:
+
+
+.. code-block:: sh
+
+ $ conda update -n root conda conda-build
+ ...
+ $ conda install -n root pyyaml
+ ...
+
+
+.. note::
+
+ Notice the application ``conda`` is in my ``${PATH}`` and therefore the
+ shell can find it easily. **Make sure you do the same**.
+
+
+Now that you're all set, just call the script
+``bob.admin/conda/conda-bootstrapy`` and pass the name of the resulting
+environment you'd like to create:
+
+
+.. code-block:: sh
+
+ $ cd <PACKAGE>
+ $ ../bob.admin/conda/conda-bootstrap.py dev
+ ...
+
+
+This will parse the conda recipe from your package and create a new conda
+environment on your conda installation called ``dev``. The environment ``dev``
+contains all build *and* test dependencies required for your package. Activate
+this environment and you're ready.
+
+.. note::
+
+ By default, our script **will include** our `conda beta channel`_ while
+ creating your environment. You may modify the file
+ ``bob.admin/conda/build-condarc`` if you'd like to include or remove
+ channels.
+
+ In this setup, we bypass your own condarc_ setup and use a stock version
+ provided with ``bob.admin``.
+
+
+Running buildout
+----------------
+
+The last step is to create a hooked-up environment so you can quickly test
+local changes to your package w/o necessarily creating a conda-package. This
+step is the easiest:
+
+
+.. code-block:: sh
+
+ $ cd <PACKAGE> #if that is not the case
+ $ source activate dev
+ $ buildout
+ ...
+
+zc.buildout_ works by modifying the load paths of scripts to find the correct
+version of your package sources from the local checkout. After running,
+buildout creates a directory called ``bin`` on your local package checkout. Use
+the applications living there to develop your package. For example, if you need
+to run the test suite:
+
+
+.. code-block:: sh
+
+ $ ./bin/nosetests -sv
+
+
+A python interpreter clone can be used to run interactive sessions:
+
+
+.. code-block:: sh
+
+ $ ./bin/python
+
+
+**Optional** Cross-development of dependencies
+----------------------------------------------
+
+From time to time, you may wish to cross-develop multiple projects at once. For
+example, you may wish to develop ``bob.bio.face``, while *also* making
+modifications to ``bob.bio.base``. In this case, you'll need to create a
+buildout recipe (i.e., a new ``.cfg``) file that instructs buildout to also
+checkout the sources for ``bob.bio.base`` while setting up the local structure
+for ``bob.bio.face``. Follow our development guide from ``bob.extension`` at
+:ref:`bob.extension` for more instructions on this step. Once your new ``.cfg``
+is ready, use it like this setup:
+
+
+.. code-block:: sh
+
+ $ buildout -c newrecipe.cfg
+
+
+.. include:: links.rst
diff --git a/doc/index.rst b/doc/index.rst
index 38da2a77d6c039258529a4b41020b1a182645134..c2a3590b9f239d7191bfaa6f3ec4f7c7f65546f8 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -1,19 +1,57 @@
.. vim: set fileencoding=utf-8 :
-.. Andre Anjos <andre.anjos@idiap.ch>
.. Mon 17 Mar 09:23:45 2014 CET
-..
-.. Copyright (C) 2011-2014 Idiap Research Institute, Martigny, Switzerland
.. _bob.buildout:
-====================
- Bob Buildout Rules
-====================
+==============
+ Bob Buildout
+==============
-.. todolist::
+This package includes an extension and a recipe to make it easy to create
+local, directory-based complete development environments for |project|
+packages. While |project| packages are distributed as conda-based installable
+packages, the use of zc.buildout_ may greaty shorten development cycles as it
+avoids the conda-build step, which can be somewhat lengthy (e.g. for packages
+with C/C++ bindings).
+zc.buildout_, unlike conda_, can *not* handle non-python-based packages. It is
+therefore a tool to quickly create deployments or test changes on
+**python-based distributable packages**. It follows that, in order to use
+zc.buildout_, packages *need* to be python-based packages. Luckily, most of
+|project| packages fit in this category, besides also being conda_ packages, so
+zc.buildout_ can be used for development purposes in this context. Although
+zc.buildout_ supports deployment, it is currently **not** used for this purpose
+in the context of |project|. Our dependence list goes beyond Python-only
+packages (e.g. ffmpeg or blitz++) and a full-stack software deployment cannot
+be easily achieved relying only on this python development tool.
-.. todo:: Write documentation of this package
+zc.buildout_ uses Python setuptools_ to create a directory structure that
+contains scripts and dependencies which can defined in a *buildout recipe*.
+Once a buildout recipe is interpreted, zc.buildout_ will make sure all listed
+dependencies are satisfied and automatically instantiate scripts with changed
+load paths that take in consideration all python packages. In order to satistfy
+dependencies, zc.buildout_ may download missing python-based packages from
+PyPI_, the Python Package Index. In the context of |project|-based package
+development, we seldomly use this feature though.
+
+Because zc.buildout_ is used for |project| package development since quite
+sometime, most packages in the |project| ecosystem include a file called
+``buildout.cfg`` that can be used by zc.buildout_ to build a quick,
+*throw-away* development environment for the current package. A common
+misconception is that one must checkout a |project| package in order to run
+buildout. That is **not** true. zc.buildout_ is an independent development
+tool that just deployes a python-based software stack based on a *recipe*. If
+the *recipe* mentions the code of a package on the current directory, the setup
+will include this. If it doesn't, then it won't. We typically don't use
+zc.buildout_ to create deployments for |project| packages - we normally defer
+this task to conda_ as it can handle non-pythonic dependencies in a much better
+and uniform way. That said, it is possible to create python-only based
+deployments using zc.buildout_ *without* being necessarily in the root
+directory of a |project| package.
+
+This document explains how to use zc.buildout_ and ``bob.buildout``'s recipe
+and extension to construct development environments for your |project|
+packages.
Documentation
@@ -22,6 +60,11 @@ Documentation
.. toctree::
:maxdepth: 2
+ guide
+ buildout
+ development
+ py_api
+
Indices and tables
------------------
diff --git a/doc/links.rst b/doc/links.rst
index 4884a0e927c6a2a088d808a2f344f75d628be256..0a638af018ab4121f9532773e7d945748111f07b 100644
--- a/doc/links.rst
+++ b/doc/links.rst
@@ -1,23 +1,25 @@
.. vim: set fileencoding=utf-8 :
-.. Andre Anjos <andre.anjos@idiap.ch>
-.. Tue 20 Mar 2012 08:57:32 CET
-..
-.. Copyright (C) 2011-2014 Idiap Research Institute, Martigny, Switzerland
+.. Tue 27 Feb 2018 09:11:11 CET
.. _bob's website: https://www.idiap.ch/software/bob
.. _bob: https://www.idiap.ch/software/bob
+.. _bob's installation: https://www.idiap.ch/software/bob/install
+.. _bob packages: https://www.idiap.ch/software/bob/packages
.. _c++: http://www2.research.att.com/~bs/C++.html
+.. _conda: https://conda.io
+.. _conda channel: https://www.idiap.ch/software/bob/conda/
+.. _conda beta channel: https://www.idiap.ch/software/bob/conda/label/beta
+.. _condarc: https://conda.io/docs/user-guide/configuration/use-condarc.html
.. _distutils: http://docs.python.org/distutils/
.. _git: http://git-scm.com/
-.. _github: http://github.com/
+.. _gitlab: https://gitlab.idiap.ch/bob/
.. _idiap: http://www.idiap.ch
.. _ipython: http://ipython.scipy.org
.. _nose: https://nose.readthedocs.org/en/latest/
.. _pep 386: http://www.python.org/dev/peps/pep-0386/
.. _python: http://www.python.org
.. _pypi: http://pypi.python.org
-.. _satellite packages: https://github.com/idiap/bob/wiki/Satellite-Packages
-.. _setuptools: http://trac.edgewall.org/wiki/setuptools
+.. _setuptools: https://setuptools.readthedocs.io
.. _sphinx: http://sphinx.pocoo.org
.. _zc.buildout: http://pypi.python.org/pypi/zc.buildout/
diff --git a/doc/nitpick-exceptions.txt b/doc/nitpick-exceptions.txt
new file mode 100644
index 0000000000000000000000000000000000000000..58a8167371ee718c6c9e9bd6083246f7725b3265
--- /dev/null
+++ b/doc/nitpick-exceptions.txt
@@ -0,0 +1 @@
+py:class zc.recipe.egg.egg.Scripts
diff --git a/doc/py_api.rst b/doc/py_api.rst
new file mode 100644
index 0000000000000000000000000000000000000000..d7724473ec709ee6ce705649b5ea9a4fdcd7b6cc
--- /dev/null
+++ b/doc/py_api.rst
@@ -0,0 +1,33 @@
+.. _bob.buildout.api:
+
+============
+ Python API
+============
+
+This section includes information for using the Python API of ``bob.buildout``.
+
+
+Extension
+---------
+
+.. automodule:: bob.buildout.extension
+
+
+Recipe
+------
+
+.. automodule:: bob.buildout.scripts
+
+
+Other Modules
+-------------
+
+.. automodule:: bob.buildout.envwrapper
+
+.. automodule:: bob.buildout.tools
+
+.. automodule:: bob.buildout.script
+
+.. automodule:: bob.buildout.python
+
+.. automodule:: bob.buildout.dbpy