templates.rst

Additional Considerations

These instructions describe how create new packages for either Bob_ or BEAT_ and provides information on how to generate a complete, but empty package from scratch.

Note

If you'd like to update part of your package setup, follow similar instructions and then copy the relevant files to your existing setup, overriding portions you know are correct.

Warning

These instructions may change as we get more experience in what needs to be changed. In case that happens, update your package by generating a new setup and copying the relevant parts to your existing package(s).

Unit tests

Writing unit tests is an important asset on code that needs to run in different platforms and a great way to make sure all is OK. Test units are run with nose_. To run the test units on your package call:

$ ./bin/nosetests -v
bob.example.library.test.test_reverse ... ok

----------------------------------------------------------------------
Ran 1 test in 0.253s

OK

This example shows the results of the tests in the bob.example.project package. Ideally, you should write test units for each function of your package ...

Note

You should put additional packages needed for testing (e.g. nosetests) in the test-requirements.txt file.

Continuous integration (CI)

Note

This is valid for people at Idiap (or external bob contributors with access to Idiap's gitlab)

Note

Before going into CI, you should make sure that your pacakge has a gitlab repository. If not, do the following in your package root folder:

$ git init
$ git remote add origin git@gitlab.idiap.ch:bob/`basename $(pwd)`
$ git add bob/ buildout.cfg COPYING doc/ MANIFEST.IN README.rst requirements.txt setup.py version.txt
$ git commit -m '[Initial commit]'
$ git push -u origin master

Copy the appropriate yml template for the CI builds:

# for pure python
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-python-only.yml > .gitlab-ci.yml
# for c/c++ extensions
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-cxx-extensions.yml | tr -d '\r' > .gitlab-ci.yml

Add the file to git:

$ git add .gitlab-ci.yml

The ci file should work out of the box. It is long-ish, but generic to any package in the system.

You also need to enable the following options - through gitlab - on your project:

1. In the project "Settings" page, make sure builds are enabled
2. If you have a private project, check the package settings and make sure that the "Deploy Keys" for our builders (all conda-* related servers) are enabled
3. Visit the "Runners" section of your package settings and enable all conda runners, for linux and macosx variants
4. Go into the "Variables" section of your package setup and add common variables corresponding to the usernames and passwords for uploading wheels and documentation tar balls to our (web DAV) server, as well as PyPI packages. You can copy required values from [the "Variables" section of bob.admin](https://gitlab.idiap.ch/bob/bob.admin/variables). N.B.: You must be logged into gitlab to access that page.
5. Make sure to disable the service "Build e-mails" (those are very annoying)
6. Setup the coverage regular expression under "CI/CD pipelines" to have the value ^TOTAL.*s+(d+%)$, which is adequate for figuring out the output of coverage report

Python package namespace

We like to make use of namespaces to define combined sets of functionality that go well together. Python package namespaces are explained in details here together with implementation details. For bob packages, we usually use the bob namespace, using several sub-namespaces such as bob.io, bob.ip, bob.learn, bob.db or (like here) bob.example.

The scripts you created should also somehow contain the namespace of the package. In our example, the script is named bob_example_project_version.py, reflecting the namespace bob.example

Distributing your work

To distribute a package, we recommend you use PyPI_. Python Packaging User Guide contains details and good examples on how to achieve this. Moreover, you can provide a conda_ package for your PyPI_ package for easier installation. In order to create a conda_ package, you need to create a conda_ recipe for that package.

For more detailed instructions on how to achieve this, please see the guidelines on bob.template.

Buildout.cfg in more details

Some notes on buildout

To be able to develop a package, we first need to build and install it locally. While developing a package, you need to install your package in development mode so that you do not have to re-install your package after every change that you do in the source. zc.buildout_ allows you to exactly do that.

Note

zc.buildout_ will create another local environment from your conda_ environment but unlike conda_ environments this environment is not isolated rather it inherits from your conda_ environment. This means you can still use the libraries that are installed in your conda_ environment. zc.buildout_ also allows you to install PyPI_ packages into your environment. You can use it to install some Python library if it is not available using conda_. Keep in mind that to install a library you should always prefer conda_ but to install your package from source in development mode, you should use zc.buildout_.

zc.buildout_ provides a buildout command. buildout takes as input a "recipe" that explains how to build a local working environment. The recipe, by default, is stored in a file called buildout.cfg. .. note:

Buildout by default looks for ``buildout.cfg`` in your current folder and
uses that configuration file. You can specify a different config file with
the ``-c`` option:

.. code:: sh

    $ buildout -c develop.cfg

Important

Once buildout runs, it creates several executable scripts in a local bin folder. Each executable is programmed to use Python from the conda environment, but also to consider (prioritarily) your package checkout. This means that you need to use the scripts from the bin folder instead of using its equivalence from your conda environment. For example, use ./bin/python instead of python.

buildout will examine the setup.py file of packages using setuptools_ and will ensure all build and run-time dependencies of packages are available either through the conda installation or it will install them locally without changing your conda environment.

The configuration file is organized in several sections, which are indicated by [], where the default section [buildout] is always required. Some of the entries need attention.

• The first entry are the eggs. In there, you can list all python packages that should be installed. These packages will then be available to be used in your environment. Dependencies for those packages will be automatically managed, as long as you keep bob.buildout in your list of extensions. At least, the current package needs to be in the eggs list.
• The extensions list includes all extensions that are required in the buildout process. By default, only bob.buildout is required, but more extensions can be added (more on that later).
• The next entry is the develop list. These packages will be installed development mode from the specified folder.

The remaining options define how the (dependent) packages are built. For example, the debug flag defined, how the C++ code in all the (dependent) packages is built. For more information refer to C/C++ modules in your package in bob.extension documentation. The verbose options handles the verbosity of the build. When the newest flag is set to true, buildout will install all packages in the latest versions, even if an older version is already available.

Note

We normally set newest = False to avoid downloading already installed dependencies. Also, it installs by default the latest stable version of the package, unless prefer-final = False, in which case the latest available on PyPI, including betas, will be installed.

Warning

Compiling packages in debug mode (debug = true) will make them very slow. You should only use this option when you are developing and not for running experiments or production.

When the buildout command is invoked it will perform the following steps:

1. It goes through the list of eggs, searched for according packages and installed them locally.
2. It populates the ./bin directory with all the console_scripts that you have specified in the setup.py.

Important

One thing to note in package development is that when you change the entry points in setup.py of a package, you need to run buildout again.

Using mr.developer

One extension that may be useful is `mr.developer`_. It allows to develop several packages at the same time. This extension will allow buildout to automatically check out packages from git repositories, and places them into the ./src directory. It can be simply set up by adding mr.developer to the extensions section.

In this case, the develop section should be augmented with the packages you would like to develop. There, you can list directories that contain Python packages, which will be build in exactly the order that you specified. With this option, you can tell buildout particularly, in which directories it should look for some packages.

[buildout]
parts = scripts

extensions = bob.buildout
             mr.developer

newest = false
verbose = true
debug = false

auto-checkout = *

develop = src/bob.extension
          src/bob.blitz

eggs = bob.extension
       bob.blitz

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

[sources]
bob.extension = git https://gitlab.idiap.ch/bob/bob.extension
bob.blitz = git https://gitlab.idiap.ch/bob/bob.blitz

A new section called [sources] appears, where the package information for `mr.developer`_ is initialized. For more details, please read its documentation. mr.developer does not automatically place the packages into the develop list (and neither in the eggs), so you have to do that yourself.

With this augmented buildout.cfg, the buildout command will perform the following steps:

1. It checks out the packages that you specified using mr.developer.
2. It develops all packages in the develop section (it links the source of the packages to your local environment).
3. It will go through the list of eggs and search for according packages in the following order:
   1. In one of the already developed directories.
   2. In the python environment, e.g., packages installed with pip.
   3. Online, i.e. on PyPI_.
4. It will populate the ./bin directory with all the console_scripts that you have specified in the setup.py. In our example, this is ./bin/bob_new_version.py.

The order of packages that you list in eggs and develop are important and dependencies should be listed first. Especially, when you want to use a private package and which not available through `pypi`_. If you do not specify them in order, you might face with some errors like this:

Could not find index page for 'a.bob.package' (maybe misspelled?)

If you see such errors, you may need to add the missing package to eggs and develop and sources (of course, respecting the order of dependencies).

Your local environment

After buildout has finished, you should now be able to execute ./bin/python. When using the newly generated ./bin/python script, you can access all packages that you have developed, including your own package:

$ ./bin/python

>>> import bob.blitz
>>> bob.blitz # should print from '.../awesome-project/src/bob.blitz/...'
<module 'bob.blitz' from 'awesome-project/src/bob.blitz/bob/blitz/__init__.py'>
>>> print(bob.blitz.get_config())
bob.blitz: 2.0.15b0 [api=0x0202] (awesome-project/src/bob.blitz)
* C/C++ dependencies:
  - Blitz++: 0.10
  - Boost: 1.61.0
  - Compiler: {'version': '4.8.5', 'name': 'gcc'}
  - NumPy: {'abi': '0x01000009', 'api': '0x0000000A'}
  - Python: 2.7.13
* Python dependencies:
  - bob.extension: 2.4.6b0 (awesome-project/src/bob.extension)
  - numpy: 1.12.1 (miniconda/envs/bob3py27/lib/python2.7/site-packages)
  - setuptools: 36.4.0 (miniconda/envs/bob3py27/lib/python2.7/site-packages)

Everything is now setup for you to continue the development of the packages. Moreover, you can learn more about |project| packages and learn to create new ones in .

Anatomy of a new package

bob.<awesome-project>
+-- bob
  +-- __init__.py #namespace init for "bob"
+-- conda
  +-- meta.yaml
+-- doc
  +-- img
  +-- conf.py
  +-- index.rst
  +-- links.rst
+-- .gitignore
+-- .gitlab-ci.yml
+-- buildout.cfg
+-- COPYING
+-- MANIFEST.IN
+-- README.rst
+-- requirements.txt
+-- setup.py
+-- version.txt

There is a folder named conda that includes a file meta.yaml. As explained earlier this files includes the information used to prepare a proper conda environment.

The folder named bob which should only include a file __init__.py at this stage is where you will put all your new code and functionality in.

The folder named doc includes the minimum necessary information for building package documentation. The file conf.py is used by sphinx to build the documentation.

.gitlab-ci.yml includes the information about building packages on the ci. We will talk about it later.

COPYING??? MANIFEST.IN???

Continuous Integration and Deployment (CI)

If you'd like just to update CI instructions, copy the file .gitlab-ci.yml from bob/devtools/templates/.gitlab-ci.yml overriding your existing one:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/templates/.gitlab-ci.yml > .gitlab-ci.yml
$ git add .gitlab-ci.yml
$ git commit -m '[ci] Updated CI instructions' .gitlab-ci.yml

The ci file should work out of the box, it is just a reference to a global configuration file that is adequate for all packages inside the Bob_/BEAT_ ecosystem.

You also remember to enable the following options on your project:

1. In the project "Settings" page, make sure builds are enabled
2. Visit the "Runners" section of your package settings and enable all runners with the docker and macosx tags.
3. Setup the coverage regular expression under "CI/CD pipelines" to have the value ^TOTAL.*s+(d+%)$, which is adequate for figuring out the output of coverage report

New unexisting dependencies

If your package depends on third-party packages (not Bob_ or BEAT_ existing resources) that are not in the CI, but exist on the conda defaults channel, you should perform some extra steps:

1. Add the package in the meta.yml file of bob-devel in bob/bob.conda/conda/bob-devel:

   requirements:
     host:
       - python {{ python }}
       - {{ compiler('c') }}
       - {{ compiler('cxx') }}
       # Dependency list of bob packages. Everything is pinned to allow for better
       # reproducibility. Please keep this list sorted. It is recommended that you
       # update all dependencies at once (to their latest version) each time you
       # modify the dependencies here. Use ``conda search`` to find the latest
       # version of packages.
       - boost 1.65.1
       - caffe 1.0  # [linux]
       - click 6.7
       - click-plugins 1.0.3
       - ..
       - [your dependency here]

2. At the same file, update the version with the current date, in the format preset.

   package:
     name: bob-devel
     version: 2018.05.02  <-- HERE

3. Update the beat-devel and bob-devel versions in the meta.yml file inside bob/bob.conda/conda/beat-devel:

   package:
     name: beat-devel
     version: 2018.05.02  <-- HERE
   
   [...]
   
   requirements:
     host:
       - python {{ python }}
       - bob-devel 2018.05.02  <-- HERE
       - requests 2.18.4

4. Update the conda_build_config.yaml in bob/bob.devtools/bob/devtools/data/conda_build_config.yaml with your dependencies, and with the updated version of bob-devel and beat-devel. See this here and this MR here for concrete examples on how to do this.

   Note

   This step should be performed after bob.conda's pipeline on master is finished (i.e. perform steps 1 to 3 in a branch, open a merge request and wait for it to be merged, and wait for the new master branch to be "green").

Conda recipe

The CI system is based on conda recipes to build the package. The recipes are located in the conda/meta.yaml file of each package. You can start to modify the recipe of each package from the template generated by bdt template command as explained above, for new packages.

The template meta.yaml file in this package is up-to-date. If you see a Bob_ or BEAT_ package that does not look similar to this recipe, please let us know as soon as possible.

You should refrain from modifying the recipe except for the places that you are asked to modify. We want to keep recipes as similar as possible so that updating all of them in future would be possible by a script.

Each recipe is unique to the package and need to be further modified by the package maintainer to work. The reference definition of the meta.yaml file is https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html. The meta.yaml file (referred to as the recipe) will contain duplicate information that is already documented in setup.py, requirements.txt, and, eventually, in test-requirements.txt. For the time being you have to maintain both the meta.yaml file and the other files.

Let's walk through the conda/meta.yaml file (the recipe) that you just created and further customize it to your package. You need to carry out all the steps below otherwise the template meta.yaml is not usable as it is.