development.rst

Local development of |project| packages

Very often, developers of |project| packages are confronted with the need to clone repositories locally and develop installation/build and runtime code. It is recommended to create isolated environments to develop new projects using conda_ and zc.buildout. Tools implemented in bob.devtools helps automate this process. In the following we talk about how to checkout and build one or several packages from their git_ source and build proper isolated environments to develop them.

TLDR

Suppose you want to develop two packages, bob.extension and bob.blitz, locally:

• Install conda_.
• Add our `conda channel`_ to your channels.

$ conda config --set show_channel_urls True
$ conda config --add channels defaults
$ conda config --add channels https://www.idiap.ch/software/bob/conda

• Install bob.devtools on a conda environment (create a new environment or install it on your base)

$ conda create -n bdt -c https://www.idiap.ch/software/bob/conda bob.devtools

• Create an empty bob package using bdt new command:

$ conda activate bdt
$ bdt new -vv bob/bob.awesome-project author_name author_email
$ cd bob.awesome-project

• Edit buildout.cfg file to include the packages you want to develop.

[buildout]
parts = scripts

develop = src/bob.extension
          src/bob.blitz
          .

eggs = bob.extension
       bob.blitz
       bob.awesome-project

extensions = bob.buildout
             mr.developer

auto-checkout = *

newest = false
verbose = true
debug = false

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

[sources]
bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
bob.blitz = git https://gitlab.idiap.ch/bob/bob.blitz
; or
; bob.extension = git git@gitlab.idiap.ch:bob/bob.extension.git
; bob.blitz = git git@gitlab.idiap.ch:bob/bob.blitz.git

• Optionally add the packages you want to have in your local installation but not developing them to eggs in the buildout.cfg file and the requirements/host section of the conda/meta.yaml file.
• Create an isolated environment for the your project.

$ bdt create --stable -vv awesome-project
$ source activate awesome-project

• Install dependencies for bob.extension and bob.blitz using conda install command. You can find out the required dependencies by looking at conda/meta.yaml file for each package.

$ conda install gcc_linux-64 gxx_linux-64 libblitz

• Run buildout and check if your desired package is being imported from the awesome-project/src folder.

$ buildout
$ ./bin/python  # you should use this python to run things from now on

>>> import bob.blitz
>>> bob.blitz # should print from '.../awesome-project/src/bob.blitz/...'
<module 'bob.blitz' from 'awesome-project/src/bob.blitz/bob/blitz/__init__.py'>
>>> print(bob.blitz.get_config())
bob.blitz: 2.0.19b0 [api=0x0202] (awesome-project/src/src/bob.blitz)
* C/C++ dependencies:
  - Blitz++: 0.10
  - Boost: 1.62.0
  - Compiler: {'name': 'gcc', 'version': '7.3.0'}
  - NumPy: {'abi': '0x01000009', 'api': '0x0000000D'}
  - Python: 3.6.9
* Python dependencies:
  - bob.extension: 3.1.3b0 (.../awesome-project/src/bob.extension)
  - click: 7.0 (.../awesome-project/eggs/Click-7.0-py3.6.egg)
  - click-plugins: 1.1.1 (.../awesome-project/eggs/click_plugins-1.1.1-py3.6.egg)
  - numpy: 1.17.2 (awesome-project/eggs/numpy-1.17.2-py3.6-linux-x86_64.egg)
  - setuptools: 41.0.1 (.../miniconda3/envs/awesome-project/lib/python3.6/site-packages)

Optionally:

• run nosetests (e.g. of bob.extension):

$ ./bin/nosetests -sv bob.extension

• build the docs (e.g. of bob.extension):

$ export BOB_DOCUMENTATION_SERVER="https://www.idiap.ch/software/bob/docs/bob/%(name)s/master/"
# or with private docs also available at Idiap. Ask for its path from colleagues.
$ export BOB_DOCUMENTATION_SERVER="https://www.idiap.ch/software/bob/docs/bob/%(name)s/master/|http://path/to/private/docs/bob/%(name)s/master/"
$ cd src/bob.extension
$ ../../bin/sphinx-build -aEn doc sphinx  # make sure it finishes without warnings.
$ firefox sphinx/index.html  # view the docs.

Note

Sometimes when you are calling a function not interactively it is not acting normally. In that case import pkg_resources before importing your package. It is a known issue and we are working on it.

$ ./bin/python -c "import pkg_resources; import bob.blitz; print(bob.blitz)"

Local development of an existing Bob package

Checking out |project| package sources

|project| packages are developed through Gitlab_. In order to checkout a package, just use git_:

$ git clone https://gitlab.idiap.ch/bob/<package>

Where <package> is the package you want to develop. Various packages exist in |project|'s gitlab_ instance.

Create an isolated conda environment

Now that we have the package checked out we need an isolated environment with proper configuration to develop the package. bob.devtools provides a tool that automatically creates such environment.

Before proceeding, you need to make sure that you already have a conda_ environment that has bob.devtools installed in. Refer to :ref:`bob.devtools.install` for information. Here we assume that you have a conda environment named bdt with installed bob.devtools

$ cd <package>
$ conda activate bdt
$ bdt create -vv dev
$ conda activate dev

Now you have an isolated conda environment with proper channels set. Add info about what bdt create does! For more information about conda channels refer to `conda channel documentation`_.

Note

When developing and testing new features, one often wishes to work against the very latest, bleeding edge, available set of changes on dependent packages.

bdt create command adds `Bob beta channels`_ to highest priority which creates an environment with the latest dependencies instead of the latest stable versions of each package. To see which channels you are using run:

$ conda config --get channels

Note

We recommend creating a new conda_ environment for every project or task that you work on. This way you can have several isolated development environments which can be very different form each other.

Running buildout

The last step is to create a hooked-up environment so you can quickly test local changes to your package w/o necessarily creating a conda-package. This step is the easiest:

$ cd <package> #if that is not the case
$ conda activate dev
$ buildout
...

zc.buildout_ works by modifying the load paths of scripts to find the correct version of your package sources from the local checkout. After running, buildout creates a directory called bin on your local package checkout. Use the applications living there to develop your package. For example, if you need to run the test suite:

$ ./bin/nosetests -sv

A python interpreter clone can be used to run interactive sessions:

$ ./bin/python

You can see what is installed in your environment:

$ conda list

And you can install new necessary packages using conda:

$ conda install <package>

Note

If you want to debug a package regarding the issues showing on the ci you can use bob.devtools. Make sure the conda environment containing bob.devtools is activated.

$ cd <package>
$ conda activate bdt
$ bdt local build

One important advantage of using conda_ and zc.buildout_ is that it does not require administrator privileges for setting up any of the above. Furthermore, you will be able to create distributable environments for each project you have. This is a great way to release code for laboratory exercises or for a particular publication that depends on |project|.

Local development of a new Bob package

It so happens that you want to develop several bob.packages against each other for your project. In this case you can make a new package using bdt commands and develop it. You need to activate your conda environment with bob.devtools installed in it.

$ conda activate bdt
$ bdt new -vv bob/bob.awesome-project author_name author_email

This command will create a new bob package named "awesome-project" that includes the correct anatomy but it doesn't have information about the packages you are depending on.

In the root of your project there is a file buildout.cfg which should look like:

[buildout]
parts = scripts
develop = .
eggs = bob.awesome-project
extensions = bob.buildout
newest = false
verbose = true

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

Let's assume you need to develop two packages at the same time, bob.extension and bob.blitz.

You need to add these packages to the buildout.cfg file in the newly created folder.

[buildout]
parts = scripts

develop = src/bob.extension
          src/bob.blitz
          .

eggs = bob.extension
       bob.blitz
       bob.awesome-project

extensions = bob.buildout
             mr.developer

auto-checkout = *
newest = false
verbose = true

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

[sources]
bob.extension = git git@gitlab.idiap.ch:bob/bob.extension.git
bob.blitz = git git@gitlab.idiap.ch:bob/bob.blitz.git

When you build your new package the dependent packages (in this example bob.extension and bob.blitz) will be checked out on folder src in the root of your project.

Before you proceed with building your package you need to make a new isolated conda environment.

$ cd bob.awesome-project
$ conda activate bdt
$ bdt create --stable -vv awesome-project
$ conda activate awesome-project

Now you have a conda environment that is using stable channels to install bob packages. Some of bob packages need compilers to be build, so before running buildout you need to install the following packages:

$ conda install gcc_linux-64 gxx_linux-64

Note

for macos you need to install the following:

$ conda install ??

Now you can run buildout as usual.

$ buildout

Note

sometimes you may need some of bob.packages available in your local directory without necessarily developing them. In that case first install it on your conda environment using conda install command. Then add it to the eggs section in your buildout.cfg file and then run buildout.

If you knew beforehand what are those packages that you need you can add them to the conda/meta.yaml file and then create the conda environment using bdt create command which installs those packages automatically.

Some notes on buildout

To be able to develop a package, we first need to build and install it locally. While developing a package, you need to install your package in development mode so that you do not have to re-install your package after every change that you do in the source. zc.buildout_ allows you to exactly do that.

Note

zc.buildout_ will create another local environment from your conda_ environment but unlike conda_ environments this environment is not isolated rather it inherits from your conda_ environment. This means you can still use the libraries that are installed in your conda_ environment. zc.buildout_ also allows you to install PyPI_ packages into your environment. You can use it to install some Python library if it is not available using conda_. Keep in mind that to install a library you should always prefer conda_ but to install your package from source in development mode, you should use zc.buildout_.

zc.buildout_ provides a buildout command. buildout takes as input a "recipe" that explains how to build a local working environment. The recipe, by default, is stored in a file called buildout.cfg. .. note:

Buildout by default looks for ``buildout.cfg`` in your current folder and
uses that configuration file. You can specify a different config file with
the ``-c`` option:

.. code:: sh

    $ buildout -c develop.cfg

Important

Once buildout runs, it creates several executable scripts in a local bin folder. Each executable is programmed to use Python from the conda environment, but also to consider (prioritarily) your package checkout. This means that you need to use the scripts from the bin folder instead of using its equivalence from your conda environment. For example, use ./bin/python instead of python.

buildout will examine the setup.py file of packages using setuptools_ and will ensure all build and run-time dependencies of packages are available either through the conda installation or it will install them locally without changing your conda environment.

The configuration file is organized in several sections, which are indicated by [], where the default section [buildout] is always required. Some of the entries need attention.

• The first entry are the eggs. In there, you can list all python packages that should be installed. These packages will then be available to be used in your environment. Dependencies for those packages will be automatically managed, as long as you keep bob.buildout in your list of extensions. At least, the current package needs to be in the eggs list.
• The extensions list includes all extensions that are required in the buildout process. By default, only bob.buildout is required, but more extensions can be added (more on that later).
• The next entry is the develop list. These packages will be installed development mode from the specified folder.

The remaining options define how the (dependent) packages are built. For example, the debug flag defined, how the C++ code in all the (dependent) packages is built. For more information refer to C/C++ modules in your package in bob.extension documentation. The verbose options handles the verbosity of the build. When the newest flag is set to true, buildout will install all packages in the latest versions, even if an older version is already available.

Note

We normally set newest = False to avoid downloading already installed dependencies. Also, it installs by default the latest stable version of the package, unless prefer-final = False, in which case the latest available on PyPI, including betas, will be installed.

Warning

Compiling packages in debug mode (debug = true) will make them very slow. You should only use this option when you are developing and not for running experiments or production.

When the buildout command is invoked it will perform the following steps:

1. It goes through the list of eggs, searched for according packages and installed them locally.
2. It populates the ./bin directory with all the console_scripts that you have specified in the setup.py.

Important

One thing to note in package development is that when you change the entry points in setup.py of a package, you need to run buildout again.

Using mr.developer

One extension that may be useful is `mr.developer`_. It allows to develop several packages at the same time. This extension will allow buildout to automatically check out packages from git repositories, and places them into the ./src directory. It can be simply set up by adding mr.developer to the extensions section.

In this case, the develop section should be augmented with the packages you would like to develop. There, you can list directories that contain Python packages, which will be build in exactly the order that you specified. With this option, you can tell buildout particularly, in which directories it should look for some packages.

[buildout]
parts = scripts

extensions = bob.buildout
             mr.developer

newest = false
verbose = true
debug = false

auto-checkout = *

develop = src/bob.extension
          src/bob.blitz

eggs = bob.extension
       bob.blitz

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

[sources]
bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
bob.blitz = git https://gitlab.idiap.ch/bob/bob.blitz

A new section called [sources] appears, where the package information for `mr.developer`_ is initialized. For more details, please read its documentation. mr.developer does not automatically place the packages into the develop list (and neither in the eggs), so you have to do that yourself.

With this augmented buildout.cfg, the buildout command will perform the following steps:

1. It checks out the packages that you specified using mr.developer.
2. It develops all packages in the develop section (it links the source of the packages to your local environment).
3. It will go through the list of eggs and search for according packages in the following order:
   1. In one of the already developed directories.
   2. In the python environment, e.g., packages installed with pip.
   3. Online, i.e. on PyPI_.
4. It will populate the ./bin directory with all the console_scripts that you have specified in the setup.py. In our example, this is ./bin/bob_new_version.py.

The order of packages that you list in eggs and develop are important and dependencies should be listed first. Especially, when you want to use a private package and which not available through `pypi`_. If you do not specify them in order, you might face with some errors like this:

Could not find index page for 'a.bob.package' (maybe misspelled?)

If you see such errors, you may need to add the missing package to eggs and develop and sources (of course, respecting the order of dependencies).

Your local environment

After buildout has finished, you should now be able to execute ./bin/python. When using the newly generated ./bin/python script, you can access all packages that you have developed, including your own package:

$ ./bin/python

>>> import bob.blitz
>>> bob.blitz # should print from '.../awesome-project/src/bob.blitz/...'
<module 'bob.blitz' from 'awesome-project/src/bob.blitz/bob/blitz/__init__.py'>
>>> print(bob.blitz.get_config())
bob.blitz: 2.0.15b0 [api=0x0202] (awesome-project/src/bob.blitz)
* C/C++ dependencies:
  - Blitz++: 0.10
  - Boost: 1.61.0
  - Compiler: {'version': '4.8.5', 'name': 'gcc'}
  - NumPy: {'abi': '0x01000009', 'api': '0x0000000A'}
  - Python: 2.7.13
* Python dependencies:
  - bob.extension: 2.4.6b0 (awesome-project/src/bob.extension)
  - numpy: 1.12.1 (miniconda/envs/bob3py27/lib/python2.7/site-packages)
  - setuptools: 36.4.0 (miniconda/envs/bob3py27/lib/python2.7/site-packages)

Everything is now setup for you to continue the development of the packages. Moreover, you can learn more about |project| packages and learn to create new ones in .