diff --git a/doc/development.rst b/doc/development.rst
new file mode 100644
index 0000000000000000000000000000000000000000..b131739aa590a51f0aee4bf95415fd649ea0ac1b
--- /dev/null
+++ b/doc/development.rst
@@ -0,0 +1,376 @@
+.. _bob.devtools.development:
+
+===============================
+ Local development of packages
+===============================
+
+Very often, developers are confronted with the need to
+clone package repositories locally and develop installation/build and runtime code.
+It is recommended to create isolated environments to develop new projects using conda_ and zc.buildout_.
+Tools implemented in `bob.devtools` helps automate this process for |project| packages. In the following we talk about how to checkout and build one or several packages from their git_ source and build proper isolated environments to develop them. Then we will describe how to create a new bob package from scratch and develop existing bob packages along side it.
+
+TLDR
+====
+
+Suppose you want to checkout the package ``bob.blitz`` from source and start developing it locally. We will use the tools implemented in ``bob.devtools`` to create a proper developing environment to build and develop ``bob.blitz``. We assume you have ``bob.devtools`` installed on a conda environment named ``bdt`` (Refer to :ref:`bob.devtools.install` for detailed information.)
+
+* Checkout the source of the package from git:
+
+.. code-block:: sh
+
+ $ git clone https://gitlab.idiap.ch/bob/bob.blitz
+
+* Create a proper conda environment:
+
+.. code-block:: sh
+
+ $ cd bob.blitz
+ $ conda activate bdt
+ $ bdt create -vv dev
+ $ conda deactivate
+ $ conda activate dev
+
+* Build the package using buildout:
+
+.. code-block:: sh
+
+ $ buildout
+ $ ./bin/python # you should use this python to run things from now on
+
+for example:
+
+.. code-block:: python
+
+ >>> import bob.blitz
+ >>> bob.blitz # should print from '.../bob.blitz/bob/blitz/...'
+ <module 'bob.blitz' from '.../bob.blitz/bob/blitz/__init__.py'>
+ >>> print(bob.blitz.get_config())
+ bob.blitz: 2.0.20b0 [api=0x0202] (.../bob.blitz)
+ * C/C++ dependencies:
+ - Blitz++: 0.10
+ - Boost: 1.67.0
+ - Compiler: {'name': 'gcc', 'version': '7.3.0'}
+ - NumPy: {'abi': '0x01000009', 'api': '0x0000000D'}
+ - Python: 3.6.9
+ * Python dependencies:
+ - bob.extension: 3.2.1b0 (.../envs/dev/lib/python3.6/site-packages)
+ - click: 7.0 (.../envs/dev/lib/python3.6/site-packages)
+ - click-plugins: 1.1.1 (.../envs/dev/lib/python3.6/site-packages)
+ - numpy: 1.16.4 (.../envs/dev/lib/python3.6/site-packages)
+ - setuptools: 41.0.1 (.../envs/dev/lib/python3.6/site-packages)
+
+* You can optionally run the test suit to check your installation:
+
+.. code-block:: sh
+
+ $ ./bin/nosetests -sv
+
+.. note::
+
+ Sometimes when you are calling a function not interactively it is not acting normally. In that case import ``pkg_resources`` before importing your package. It is a known issue and we are working on it.
+
+ .. code-block:: sh
+
+ $ ./bin/python -c "import pkg_resources; import bob.blitz; print(bob.blitz)"
+
+
+.. bob.devtools.local_development:
+
+Local development of existing packages
+======================================
+To develop existing |project| packages you need to checkout their source code and create a proper development environment using `buildout`.
+
+
+Checking out package sources
+----------------------------
+|project| packages are developed through gitlab_. Various packages exist
+in |project|'s gitlab_ instance. In the following we assume you want to install and build locally the ``bob.blitz`` package. In order to checkout a package, just use git_:
+
+
+.. code-block:: sh
+
+ $ git clone https://gitlab.idiap.ch/bob/bob.blitz
+
+
+Create an isolated conda environment
+------------------------------------
+
+Now that we have the package checked out we need an isolated environment with proper configuration to develop the package. ``bob.devtools`` provides a tool that automatically creates such environment.
+Before proceeding, you need to make sure that you already have a conda_ environment with ``bob.devtools`` installed in it (Refer to :ref:`bob.devtools.install` for more information). let's assume that you have a conda environment named ``bdt`` with installed ``bob.devtools``.
+
+.. code-block:: sh
+
+ $ cd bob.blitz
+ $ conda activate bdt
+ $ bdt create -vv dev
+ $ conda deactivate
+ $ conda activate dev
+
+
+Now you have an isolated conda environment named `dev` with proper channels set. For more information about conda channels refer to `conda channel documentation`_.
+
+The `bdt create` command assumes a directory named `conda`, exists on the current directory. The `conda` directory contains a file named `meta.yaml`, that is the recipe required to create a development environment for the package you want to develop.
+
+.. note::
+
+ When developing and testing new features, one often wishes to work against the very latest, *bleeding edge*, available set of changes on dependent packages.
+
+ `bdt create` command adds `Bob beta channels`_ to highest priority which creates an environment with the latest dependencies instead of the latest *stable* versions of each package.
+
+ If you want to create your environment using *stable* channels, you can use this command instead:
+
+ .. code-block:: sh
+
+ $ bdt create --stable -vv dev
+
+ To see which channels you are using run:
+
+ .. code-block:: sh
+
+ $ conda config --get channels
+
+
+
+.. note::
+
+ We recommend creating a new conda_ environment for every project or task
+ that you work on. This way you can have several isolated development
+ environments which can be very different form each other.
+
+
+Running buildout
+----------------
+
+The last step is to create a hooked-up environment so you can quickly test
+local changes to your package w/o necessarily creating a conda-package.
+zc.buildout_ takes care of it by modifying the load paths of scripts to find the correct
+version of your package sources from the local checkout. It by default uses a file named `buildout.cfg`, in the package directory. For our example package it looks like:
+
+.. code-block:: ini
+
+ ; vim: set fileencoding=utf-8 :
+ ; Mon 08 Aug 2016 14:33:54 CEST
+
+ [buildout]
+ parts = scripts
+ develop = .
+ eggs = bob.blitz
+ extensions = bob.buildout
+ newest = false
+ verbose = true
+
+ [scripts]
+ recipe = bob.buildout:scripts
+
+To find our more information about different section of this file, refer to :ref:`bob.devtools.buildout`.
+
+Now you just need to run buildout:
+
+.. code-block:: sh
+
+ $ cd bob.blitz #if that is not the case
+ $ conda activate dev #if that is not the case
+ $ buildout
+
+
+
+After running, buildout creates a directory called ``bin`` on your local package checkout. Use
+the applications living there to develop your package. For example, if you need
+to run the test suite:
+
+
+.. code-block:: sh
+
+ $ ./bin/nosetests -sv
+
+or build the documentation:
+
+.. code-block:: sh
+
+ $./bin/sphinx-build -aEn doc sphinx # make sure it finishes without warnings.
+ $ firefox sphinx/index.html # view the docs.
+
+.. note::
+
+ `buildout` by default uses the file `buildout.cfg` but you can specify another file by using -c option. In fact for developing packages especially if they need to be developed along with other packages, another file, namely `develop.cfg` is used like following:
+
+ .. code-block:: sh
+
+ $ buildout -c develop.cfg
+
+
+A python interpreter clone can be used to run interactive sessions:
+
+.. code-block:: sh
+
+ $ ./bin/python
+
+You can see what is installed in your environment:
+
+.. code-block:: sh
+
+ $ conda list
+
+And you can install new packages using conda:
+
+.. code-block:: sh
+
+ $ conda install <package>
+
+.. note::
+
+ If you want to debug a package regarding an issues showing on the ci you can use `bob.devtools`. Make sure the conda environment containing `bob.devtools` is activated.
+
+ .. code-block:: sh
+
+ $ cd <package>
+ $ conda activate bdt
+ $ bdt local build
+
+
+One important advantage of using conda_ and zc.buildout_ is that it does
+**not** require administrator privileges for setting up any of the above.
+Furthermore, you will be able to create distributable environments for each
+project you have. This is a great way to release code for laboratory exercises
+or for a particular publication that depends on |project|.
+
+Developing multiple existing packages simultaneously
+----------------------------------------------------
+It so happens that you want to develop several packages against each other for your project. Let's assume you want to develop ``bob.blitz`` and ``bob.extension`` simultaneously. ``bob.blitz`` is dependent on ``bob.devtools``. First we checkout package ``bob.blitz`` and build an isolated conda environment as explained in the previous section. Then edit `buildout.cfg` file (or `develop.cfg`) and add ``bob.extension`` to it as following:
+
+
+.. code-block:: ini
+
+ [buildout]
+ parts = scripts
+
+ develop = src/bob.extension
+ .
+
+ eggs = bob.blitz
+
+ extensions = bob.buildout
+ mr.developer
+
+ auto-checkout = *
+
+
+ ; options for bob.buildout extension
+ debug = true
+ verbose = true
+ newest = false
+
+ [sources]
+ bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
+
+ [scripts]
+ recipe = bob.buildout:scripts
+
+
+Now you can run `buildout` as usual. The ``bob.extension`` will be checked out on `src` folder on the root of your project.
+
+.. note::
+
+ The flag `debug = true` is usually used when in development mode.
+
+
+.. _bob.devtools.create_package:
+
+Local development of a new package
+==================================
+
+In this section we explain how to create a new bob package from scratch and start developing it. Once again ``bob.devtools`` is here to help you. You need to activate your conda environment with ``bob.devtools`` installed in it.
+
+.. code-block:: sh
+
+ $ conda activate bdt
+ $ bdt new -vv bob/bob.project.awesome author_name author_email
+
+This command will create a new bob package named "bob.project.awesome" that includes the correct anatomy of a package. For more information about the functionality of each file check :ref:`bob.devtools.anatomy`.
+
+In the root of your project there is a file `buildout.cfg` used by `buildout` to build your package locally. It should look like:
+
+.. code-block:: ini
+
+ [buildout]
+ parts = scripts
+ develop = .
+ eggs = bob.project.awesome
+ extensions = bob.buildout
+ newest = false
+ verbose = true
+
+ [scripts]
+ recipe = bob.buildout:scripts
+ dependent-scripts = true
+
+
+Now you have all the necessary tools available and you can make a development environment using `bdt create` command, run `buildout` in it and start developing your package.
+
+.. code-block:: sh
+
+ $ cd bob.project.awesome
+ $ conda activate bdt
+ $ bdt create --stable -vv awesome-project #here we used the stable channels to make the conda environment.
+ $ conda activate awesome-project
+ $ buildout
+
+
+Developing existing bob packages along with your new package
+------------------------------------------------------------
+
+Let's assume you need to develop two packages, ``bob.extension`` and ``bob.blitz``, as part of developing your new ``bob.project.awesome`` package.
+
+You need to add these packages to the ``buildout.cfg`` file in the newly created folder.
+
+.. code-block:: ini
+
+ [buildout]
+ parts = scripts
+
+ develop = src/bob.extension
+ src/bob.blitz
+ .
+
+ eggs = bob.extension
+ bob.blitz
+ bob.project.awesome
+
+ extensions = bob.buildout
+ mr.developer
+
+ auto-checkout = *
+ newest = false
+ verbose = true
+
+ [scripts]
+ recipe = bob.buildout:scripts
+ dependent-scripts = true
+
+ [sources]
+ bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
+ bob.blitz = git https://gitlab.idiap.ch/bob/bob.blitz
+ ; or
+ ; bob.extension = git git@gitlab.idiap.ch:bob/bob.extension.git
+ ; bob.blitz = git git@gitlab.idiap.ch:bob/bob.blitz.git
+
+
+When you build your new package the dependent packages (in this example ``bob.extension`` and ``bob.blitz``) will be checked out on folder `src` in the root of your project.
+
+As usual, first create an isolated conda environment using `bdt create` command. Some of bob packages need dependencies that might not be installed on your environment. You can find these dependencies by checking `conda/meta.yaml` of each package. Install the required packages and then run buildout as usual. For our example you need to do the following:
+
+.. code-block:: sh
+
+ $ conda install gcc_linux-64 gxx_linux-64 libblitz
+ $ buildout
+
+.. note::
+
+ Sometimes you may need some of bob packages available in your local `bin` directory without necessarily developing them.
+
+ If you knew beforehand what are those packages, you can add them to "requirements/host" section of the `conda/meta.yaml` file and then create a conda environment using `bdt create`. Like this, those packages will be installed automatically. Otherwise, if you already have your conda environment, install them using `conda install` command.
+
+ When done, add those packages to the `eggs` section in your `buildout.cfg` file and then run `buildout`.
+
+
+.. include:: links.rst
diff --git a/doc/index.rst b/doc/index.rst
index c5c9d035d96cbd0abd5e2e55cab79a854209585b..3b4c2068ac9fdba8cd43ea1333382f648d56239e 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -16,8 +16,9 @@ Documentation
:maxdepth: 2
install
- release
+ development
templates
+ release
api
ci
diff --git a/doc/install.rst b/doc/install.rst
index 4903bd8801b28a0acfe216158f0e3cc8e03a20a6..d2da4d9456875c48e66318555b45355340093244 100644
--- a/doc/install.rst
+++ b/doc/install.rst
@@ -61,5 +61,20 @@ We recommend you set ``chmod 600`` to this file to avoid prying eyes to read
out your personal token. Once you have your token set up, communication should
work transparently between the built-in gitlab client and the server.
+If you would like to use the WebDAV interface to our web service for manually
+uploading contents, you may also setup the address, username and password for
+that server inside the file ``~/.bdtrc``. Here is a skeleton:
+
+.. code-block:: ini
+
+
+ [webdav]
+ server = http://example.com
+ username = username
+ password = password
+
+You may obtain these parameters from our internal page explaining the `WebDAV
+configuration`_. You shoul also set ``chmod 600`` to this file for obvious
+security reasons.
.. include:: links.rst
diff --git a/doc/links.rst b/doc/links.rst
index 85e0033ab30e2410ae0d471fe7976c48d270db3d..b933ddf12a519e0c2539c240f1747f227c894ae6 100644
--- a/doc/links.rst
+++ b/doc/links.rst
@@ -8,3 +8,15 @@
.. _shell executor: https://docs.gitlab.com/runner/executors/shell.html
.. _gitlab runner: https://docs.gitlab.com/runner/install/osx.html
.. _docker for mac: https://docs.docker.com/docker-for-mac/install/
+.. _conda channel documentation: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html
+.. _Bob beta channels: https://www.idiap.ch/software/bob/conda/label/beta
+.. _git: http://git-scm.com/
+.. _gitlab: https://gitlab.idiap.ch/bob/
+.. _nose: https://nose.readthedocs.org/en/latest/
+.. _python: http://www.python.org
+.. _pypi: http://pypi.python.org
+.. _setuptools: https://setuptools.readthedocs.io
+.. _zc.buildout: http://www.buildout.org/
+.. _mr.developer: http://pypi.python.org/pypi/mr.developer/
+.. _conda channel: https://www.idiap.ch/software/bob/conda/
+.. _webdav configuration: https://gitlab.idiap.ch/bob/private/wikis/how-to-upload-resources
diff --git a/doc/templates.rst b/doc/templates.rst
index 17d864a31587ab581ac81253059ab444a2d39735..8574dc47b68edaca807e973461551abeaf6feb2d 100644
--- a/doc/templates.rst
+++ b/doc/templates.rst
@@ -2,13 +2,11 @@
.. _bob.devtools.templates:
-==========================
- New Package Instructions
-==========================
+===========================
+ Additional Considerations
+===========================
-These instructions describe how create new packages for either Bob_ or BEAT_
-and provides information on how to generate a complete, but empty package from
-scratch.
+These instructions describes some steps that needs to be noted after creating new packages for either Bob_ or BEAT_ to incorporate the package properly in the ecosystem.
.. note::
@@ -24,15 +22,362 @@ scratch.
setup and copying the relevant parts to your existing package(s).
-Create a new package
---------------------
+Unit tests
+==========
+
+Writing unit tests is an important asset on code that needs to run in different platforms and a great way to make sure all is OK.
+Test units are run with nose_.
+To run the test units on your package call:
+
+.. code-block:: sh
+
+ $ ./bin/nosetests -v
+ bob.example.library.test.test_reverse ... ok
+
+ ----------------------------------------------------------------------
+ Ran 1 test in 0.253s
+
+ OK
+
+This example shows the results of the tests in the ``bob.example.project`` package. Ideally, you should
+write test units for each function of your package ...
+
+.. note::
+
+ You should put additional packages needed for testing (e.g. ``nosetests``)
+ in the ``test-requirements.txt`` file.
+
+
+Continuous integration (CI)
+===========================
+
+.. note::
+
+ This is valid for people at Idiap (or external bob contributors with access to Idiap's gitlab)
+
+.. note::
+
+ Before going into CI, you should make sure that your pacakge has a gitlab repository.
+ If not, do the following in your package root folder:
+
+ .. code-block:: sh
+
+ $ git init
+ $ git remote add origin git@gitlab.idiap.ch:bob/`basename $(pwd)`
+ $ git add bob/ buildout.cfg COPYING doc/ MANIFEST.IN README.rst requirements.txt setup.py version.txt
+ $ git commit -m '[Initial commit]'
+ $ git push -u origin master
+
+
+Copy the appropriate yml template for the CI builds:
+
+
+.. code-block:: sh
+
+ # for pure python
+ $ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-python-only.yml > .gitlab-ci.yml
+ # for c/c++ extensions
+ $ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-cxx-extensions.yml | tr -d '\r' > .gitlab-ci.yml
+
+Add the file to git:
+
+.. code-block:: sh
+
+ $ git add .gitlab-ci.yml
+
+
+The ci file should work out of the box. It is long-ish, but generic to any
+package in the system.
+
+You also need to enable the following options - through gitlab - on your project:
+
+1. In the project "Settings" page, make sure builds are enabled
+2. If you have a private project, check the package settings and make sure that
+ the "Deploy Keys" for our builders (all `conda-*` related servers) are
+ enabled
+3. Visit the "Runners" section of your package settings and enable all conda
+ runners, for linux and macosx variants
+4. Go into the "Variables" section of your package setup and **add common
+ variables** corresponding to the usernames and passwords for uploading
+ wheels and documentation tar balls to our (web DAV) server, as well as PyPI
+ packages. You can copy required values from [the "Variables" section of
+ bob.admin](https://gitlab.idiap.ch/bob/bob.admin/variables). N.B.: You
+ **must** be logged into gitlab to access that page.
+5. Make sure to **disable** the service "Build e-mails" (those are very
+ annoying)
+6. Setup the coverage regular expression under "CI/CD pipelines" to have the
+ value `^TOTAL.*\s+(\d+\%)$`, which is adequate for figuring out the output
+ of `coverage report`
+
+
+Python package namespace
+========================
+
+We like to make use of namespaces to define combined sets of functionality that go well together.
+Python package namespaces are `explained in details here <http://peak.telecommunity.com/DevCenter/setuptools#namespace-package>`_ together with implementation details.
+For bob packages, we usually use the ``bob`` namespace, using several sub-namespaces such as ``bob.io``, ``bob.ip``, ``bob.learn``, ``bob.db`` or (like here) ``bob.example``.
+
+
+The scripts you created should also somehow contain the namespace of the package. In our example,
+the script is named ``bob_example_project_version.py``, reflecting the namespace ``bob.example``
+
+
+Distributing your work
+======================
+
+To distribute a package, we recommend you use PyPI_.
+`Python Packaging User Guide <https://packaging.python.org/>`_ contains details
+and good examples on how to achieve this.
+Moreover, you can provide a conda_ package for your PyPI_ package for easier
+installation. In order to create a conda_ package, you need to create a conda_
+recipe for that package.
+
+.. note::
+
+ For more detailed instructions on how to distribute your packages at Idiap, please see the
+ guidelines on `Publishing Reproducible Papers at Idiap <https://gitlab.idiap.ch/bob/bob/wikis/Publishing-Reproducible-Papers-at-Idiap>`_.
+
+
+.. _bob.devtools.buildout:
+
+buildout.cfg in more details
+============================
+This section briefly explains the different entries in ``buildout.cfg`` file. For better understanding of buildout refer to its
+`documentation <http://www.buildout.org>`_
+
+
+To be able to develop a package, we first need to build and install it locally.
+While developing a package, you need to install your package in *development*
+mode so that you do not have to re-install your package after every change that
+you do in the source. zc.buildout_ allows you to exactly do that.
+
+.. note::
+ zc.buildout_ will create another local environment from your conda_
+ environment but unlike conda_ environments this environment is not isolated
+ rather it inherits from your conda_ environment. This means you can still
+ use the libraries that are installed in your conda_ environment.
+ zc.buildout_ also allows you to install PyPI_ packages into your
+ environment. You can use it to install some Python library if it is not
+ available using conda_. Keep in mind that to install a library you should
+ always prefer conda_ but to install your package from source in
+ *development* mode, you should use zc.buildout_.
+
+zc.buildout_ provides a ``buildout`` command. ``buildout`` takes as input a
+"recipe" that explains how to build a local working environment. The recipe, by
+default, is stored in a file called ``buildout.cfg``.
+
+
+.. important::
+ Once ``buildout`` runs, it creates several executable scripts in a local
+ ``bin`` folder. Each executable is programmed to use Python from the conda
+ environment, but also to consider (prioritarily) your package checkout.
+ This means that you need to use the scripts from the ``bin`` folder instead
+ of using its equivalence from your conda environment. For example, use
+ ``./bin/python`` instead of ``python``.
+
+``buildout`` will examine the ``setup.py`` file of packages using setuptools_
+and will ensure all build and run-time dependencies of packages are available
+either through the conda installation or it will install them locally without
+changing your conda environment.
+
+The configuration file is organized in several *sections*, which are indicated
+by ``[]``, where the default section ``[buildout]`` is always required. Some of
+the entries need attention.
+
+* The first entry are the ``eggs``. In there, you can list all python packages
+ that should be installed. These packages will then be available to be used in
+ your environment. Dependencies for those packages will be automatically
+ managed, **as long as you keep** ``bob.buildout`` **in your list of**
+ ``extensions``. At least, the current package needs to be in the ``eggs``
+ list.
+
+* The ``extensions`` list includes all extensions that are required in the
+ buildout process. By default, only ``bob.buildout`` is required, but more
+ extensions can be added.
+
+* The next entry is the ``develop`` list. These packages will be installed
+ *development mode* from the specified folder.
+
+The remaining options define how the (dependent) packages are built. For
+example, the ``debug`` flag defined, how the *C++ code* in
+all the (dependent) packages is built. For more information refer to *C/C++ modules in your package* in `bob.extension <https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/index.html>`_ documentation. The ``verbose`` options handles the
+verbosity of the build. When the ``newest`` flag is set to ``true``, buildout
+will install all packages in the latest versions, even if an older version is
+already available.
+
+.. note::
+
+ We normally set ``newest = False`` to avoid downloading already installed
+ dependencies. Also, it installs by default the latest stable version of the
+ package, unless ``prefer-final = False``, in which case the latest
+ available on PyPI, including betas, will be installed.
-To create a new package, just use the command ``bdt new``. Use its ``--help``
-to get more information about options you can provide.
+.. warning::
+
+ Compiling packages in debug mode (``debug = true``) will make them very
+ slow. You should only use this option when you are developing and not for
+ running experiments or production.
+
+When the buildout command is invoked it will perform the following steps:
+
+1. It goes through the list of ``eggs``, searched for according packages and
+ installed them *locally*.
+2. It populates the ``./bin`` directory with all the ``console_scripts`` that
+ you have specified in the ``setup.py``.
+
+.. important::
+
+ One thing to note in package development is that when you change the entry
+ points in ``setup.py`` of a package, you need to run ``buildout`` again.
+
+
+.. _bob.devtools.mr.developer:
+
+Using mr.developer
+------------------
+
+One extension that may be useful is `mr.developer`_. It allows to develop
+*several packages* at the same time. This extension will allow
+buildout to automatically check out packages from git repositories, and places
+them into the ``./src`` directory. It can be simply set up by adding
+``mr.developer`` to the extensions section.
+
+In this case, the develop section should be augmented with the packages you
+would like to develop. There, you can list directories that contain Python
+packages, which will be build in exactly the order that you specified. With
+this option, you can tell buildout particularly, in which directories it should
+look for some packages.
+
+.. code-block:: ini
+
+ [buildout]
+ parts = scripts
+
+ extensions = bob.buildout
+ mr.developer
+
+ newest = false
+ verbose = true
+ debug = false
+
+ auto-checkout = *
+
+ develop = src/bob.extension
+ src/bob.blitz
+
+ eggs = bob.extension
+ bob.blitz
+
+ [scripts]
+ recipe = bob.buildout:scripts
+ dependent-scripts = true
+
+ [sources]
+ bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
+ bob.blitz = git https://gitlab.idiap.ch/bob/bob.blitz
+
+A new section called ``[sources]`` appears, where the package information for
+`mr.developer`_ is initialized. For more details, please read `its
+documentation <https://pypi.python.org/pypi/mr.developer>`_. mr.developer does
+not automatically place the packages into the ``develop`` list (and neither in
+the ``eggs``), so you have to do that yourself.
+
+With this augmented ``buildout.cfg``, the ``buildout`` command will perform the
+following steps:
+
+
+
+1. It checks out the packages that you specified using ``mr.developer``.
+
+2. It develops all packages in the ``develop`` section
+ (it links the source of the packages to your local environment).
+
+3. It will go through the list of ``eggs`` and search for according packages
+ in the following order:
+
+ #. In one of the already developed directories.
+ #. In the python environment, e.g., packages installed with ``pip``.
+ #. Online, i.e. on PyPI_.
+
+4. It will populate the ``./bin`` directory with all the ``console_scripts``
+ that you have specified in the ``setup.py``. In our example, this is
+ ``./bin/bob_new_version.py``.
+
+The order of packages that you list in ``eggs`` and ``develop`` are important
+and dependencies should be listed first. Especially, when you want to use a
+private package and which not available through `pypi`_. If you do not specify
+them in order, you might face with some errors like this::
+
+ Could not find index page for 'a.bob.package' (maybe misspelled?)
+
+If you see such errors, you may need to add the missing package to ``eggs`` and
+``develop`` and ``sources`` (**of course, respecting the order of
+dependencies**).
+
+.. _bob.devtools.anatomy:
+
+Anatomy of a new package
+========================
+A typical package have the following structure:
+
+
+.. code-block:: text
+
+ .
+ +-- bob # python package (a.k.a. "the code")
+ | +-- project
+ | | +-- awesome # your code will go into this folder
+ | | | +-- __init__.py # name space init for "awesome"
+ | | +-- __init__.py # name space init for "project"
+ | +-- __init__.py # name space init for "bob"
+ +-- conda
+ | +-- meta.yaml # recipe for preparing the conda environment
+ +-- doc # documentation directory
+ | +-- img
+ | +-- conf.py # sphinx configuration
+ | +-- index.rst # documentation starting point for Sphinx
+ | +-- links.rst
+ +-- .gitignore # some settings for git
+ +-- .gitlab-ci.yml # instruction for ci integration
+ +-- buildout.cfg # buildout configuration
+ +-- COPYING # license information
+ +-- MANIFEST.IN # extras to be installed, besides the Python files
+ +-- README.rst # a minimal description of the package, in reStructuredText format
+ +-- requirements.txt # requirements of your package
+ +-- setup.py # installation instruction for this particular package
+ +-- version.txt # the (current) version of your package
+
+
+A quick overview of these files:
+
+
+bob: It is the directory that includes the source code and scripts for your package. The files should be organized in subdirectories that matches the name of your package.
+
+conda: It is the directory includes the recipe (``meta.yaml``) for preparing the base conda environment used for package development.
+
+doc: This is the directory including the minimum necessary information for building package documentation. The file `conf.py` is used by sphinx to build the documentation.
+
+.gitignore: This file includes some settings for git.
+
+.gitlab-ci.yml: It is the file including the information about building packages on the CI.
+
+buildout.cfg: This file contains the basic recipe to create a working environment for developing the package.
+
+COPYING: The file including the licensing information.
+
+MANIFEST.IN: This file contains the list of non python files and packages that are needed to be installed for your package.
+
+README.rst: It includes basic information about the package, in reStructuredText format.
+
+requirements.txt: This file contains the direct dependencies of the package.
+
+setup.py: This file contains the python packaging instructions. For detailed information refer to `setuptools`_.
+
+version.txt: The file shows the current version of the package.
Continuous Integration and Deployment (CI)
-------------------------------------------
+==========================================
If you'd like just to update CI instructions, copy the file ``.gitlab-ci.yml``
from ``bob/devtools/templates/.gitlab-ci.yml`` **overriding** your existing
@@ -61,7 +406,7 @@ You also remember to enable the following options on your project:
New unexisting dependencies
----------------------------
+===========================
If your package depends on **third-party packages** (not Bob_ or BEAT_ existing
resources) that are not in the CI, but exist on the conda ``defaults`` channel,
@@ -132,7 +477,7 @@ you should perform some extra steps:
Conda recipe
-------------
+============
The CI system is based on conda recipes to build the package. The recipes are
located in the ``conda/meta.yaml`` file of each package. You can start
@@ -161,7 +506,7 @@ the steps below otherwise the template ``meta.yaml`` is not usable as it is.
Entry-points in the ``build`` section
-=====================================
+-------------------------------------
You need to check if your package has any ``console_scripts``. These are
documented in ``setup.py`` of each package. You need to list the
@@ -205,7 +550,7 @@ recipe.
Build and host dependencies
-===========================
+---------------------------
This part of the recipe lists the packages that are required during build time
(`information on conda package requirements here
@@ -263,7 +608,7 @@ recipes. Here are some notes:
Runtime dependencies
-====================
+--------------------
In the ``requirements / run`` section of the conda recipe, you will list
dependencies that are needed when a package is used (run-time) dependencies.
@@ -348,7 +693,7 @@ Here is a list of packages that we know that they have ``run_exports``:
Testing entry-points
-====================
+--------------------
If you listed some of your ``setup.py`` ``console_sripts`` in the ``build / entry_points`` section of the conda recipe, it is adviseable you test these. For
example, if you had the examples entry points above, you would test them like:
@@ -364,7 +709,7 @@ example, if you had the examples entry points above, you would test them like:
Test-time dependencies
-======================
+----------------------
You need to list the packages here that are required during **test-time only**.
By default, add some packages. Do not remove them. The test-time dependencies
@@ -386,7 +731,7 @@ that is not committed to the project repository by mistake.
Database packages and packages with extra data
-----------------------------------------------
+==============================================
Sometimes databases or other packages require an extra download command after
installation. If this extra data is downloaded from Idiap severs, you can
@@ -404,7 +749,7 @@ packages typically require this download command to be added in the
Licensing
----------
+=========
There are 2 possible cases for the majority of packages in our ecosystem:
@@ -421,7 +766,7 @@ More info about Idiap's `open-source policy here
Headers
--------
+=======
Sometimes people add headers with licensing terms to their files. You should
inspect your library to make sure you don't have those. The Idiap TTO says this
@@ -442,14 +787,14 @@ go through all the headers.
The ``setup.py`` file
----------------------
+=====================
The ``setup.py`` should be changed to include eventual ``entry_points`` you
also included in the ``conda/meta.yaml``. We cannot guess these.
Buildout
---------
+========
The default buildout file ``buildout.cfg`` should buildout from the installed
distribution (use ``bdt create`` for that purpose) and **avoid mr.developer
@@ -459,7 +804,7 @@ provided by this package takes care of this.
The ``README.rst`` file
------------------------
+=======================
You should make the README smaller and easier to maintain. As of today, many
packages contain outdated installation instructions or outdated links. More
@@ -471,7 +816,7 @@ generation. Make it short, a single phrase is the most common size.
Sphinx documentation
---------------------
+====================
Sphinx documentation configuration goes to a file named ``doc/conf.py``. The
file ``doc/index.rst`` is the root of the documentation for your package.
@@ -504,7 +849,7 @@ text files to be placed inside the ``doc/`` directory, alongside the ``conf.py``
Project logo and branding
--------------------------
+=========================
In the gitlab Settings / General page of your project, update the logo to use
one of ours: