New Package Instructions

These instructions describe how to migrate or setup your package to gitlab and new licensing terms. At the same time, we profit to update and fix files touched by this procedure. You don't have to do all these steps at once. If you choose to only do some, prioritise by the order in this text (first do the continuous integration changes, then the licensing checks, etc).

Notice: This text may change as we get more experience in what needs to be changed. We may also automatise some of the actions below. We'll keep you posted in this case and update these instructions.

0. Set the right origin

If you haven't checked your repository from gitlab yet, you don't have to remove it and re-clone it, you can just set its origin to point to gitlab:

$ git remote set-url origin git@gitlab.idiap.ch:bob/`basename $(pwd)`
$ git pull

1. Continuous Integration

Copy the stock yml template for the CI builds:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/gitlab-ci.yml > .gitlab-ci.yml

Add the file to git:

$ git add .gitlab-ci.yml

The ci file should work out of the box. It is long-ish, but generic to any package in the system.

You also need to enable the following options on your project:

1. In the project "Settings" page, make sure builds are enabled
2. If you have a private project, check the package settings and make sure that the "Deploy Keys" for our builders (runner/beat-macosx and bpr- facedemo) are enabled.
3. Visit the "Runners" section of your package settings and enable all runners with the docker and macosx tags.
4. Setup the coverage regular expression under "CI/CD pipelines" to have the value ^TOTAL.*\s+(\d+\%)$, which is adequate for figuring out the output of coverage report

1.5 Conda recipe

The CI system is based on conda recipes to build the package. The recipes must be located in the conda folder of each package. You can start modifying the recipe of each package by copying the template recipe:

$ mkdir -p conda
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/meta.yaml > conda/meta.yaml
$ sed -i "s/<PACKAGE>/`basename $(pwd)`/g" conda/meta.yaml

The template meta.yaml file in bob.admin is up-to-date. If you see a bob package that does not look similar to this recipe, please let us know ASAP.

You should refrain from modifying the recipe except for the places that you are asked to modify. We want to keep recipes as similar as possible so that updating all of them in future would be possible by a script.

Each recipe is unique to the package and need to be further modified by the package maintainer to work. The reference definition of the meta.yaml file is here: https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html The meta.yaml file (referred to as the recipe) will contain duplicate information that is already documented in setup.py, requirements.txt, and test-requirements.txt. For the time being you have to maintain both the meta.yaml file and the other files. We have plans to remove this duplication in future: bob.extension#38 (closed) You are welcome to contribute to that.

Let's walk through the meta.yaml file (the recipe) that you just have downloaded and further customize it to our package. You need to carry out all the steps below otherwise the template meta.yaml is not usable as it is.

ENTRY_POINTS

You need to check if your package has any console_scripts. These are documented in setup.py of each package. You need to list the console_scripts entry points (only console_scripts; other entry points should not be listed in meta.yaml.) in the build section of the recipe.

• If there are no console_scripts, remove the line (line 9) that says: <ENTRY_POINTS>.

• If there are some, list them in the meta.yaml file as well: https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#python-entry-points For example, if the setup.py file contains:

      entry_points={
        'console_scripts': [
          'jman = gridtk.script.jman:main',
          'jgen = gridtk.script.jgen:main',
        ]

  You would replace <ENTRY_POINTS> with:

  build: # this line is already in the recipe. Do not add. Tabbing (two sapces) is important.
    entry_points:
      - jman = gridtk.script.jman:main
      - jgen = gridtk.script.jgen:main

BUILD_DEPS and HOST_DEPS in requirements.

This part of the recipe lists the packages that are required during build time: https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#requirements-section Having build and host requirements separately enables cross-compiling of the recipes.

• If the packages does not contain C/C++ code, remove the line that says (line 21) <BUILD_DEPS>

• Otherwise, you need to replace <BUILD_DEPS> with:

  requirements: # this line is already in the recipe. Do not add. Tabbing (two sapces) is important.
    build:
      - {{ compiler('c') }}
      - {{ compiler('cxx') }}
      - pkg-config {{ pkg_config }}
      - cmake {{ cmake }}

  The pkg-config and cmake lines are optional. If the package uses them, you need to list them.

• List all the packages that are in requirements.txt in the requirements:host section by removing <HOST_DEPS> and adding a new line per dependency. For example, here is what bob.measure has in its host:

  host: # this line is already in the recipe. Do not add. Tabbing (two sapces) is important.
    - python {{ python }} # this line is already in the recipe. Do not add.
    - setuptools {{ setuptools }} # this line is already in the recipe. Do not add.
    - bob.extension
    - bob.blitz
    - bob.core
    - bob.math
    - bob.io.base
    - matplotlib {{ matplotlib }}
    - libblitz {{ libblitz }}
    - boost {{ boost }}
    - numpy {{ numpy }}
    - docopt {{ docopt }}

  You need to add a jinja variable like {{ dependency }} in front of the dependencies that we do not develop. The jinja variable name should not contain . or -; replace those with _. The bob packages (and gridtk) should be listed as is.

• Unlike pip, conda is not limited to Python programs. If the package depends on some non-python package (like boost), you need to list it in the host section.

RUN_DEPS

In the requirements:run section, you will list dependencies that are needed when a package is used (run-time) dependencies. Usually, for pure-Python packages you list the same packages as in the host section also in the run section. This is simple BUT conda build 3 introduced a new concept named run_exports https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#pin-downstream which makes this complicated. In summary, you put all the run-time dependencies in the requirements:run section UNLESS this dependency was listed in the host section and the dependency has a run_exports. The problem is that you cannot easily find which packages actually do have run_exports unless you look at their conda recipe. Usually, all the C/C++ libraries like jpeg, hdf5 have run_exports (boost does not have one for example). All bob packages have this too. For example, here is what is inside the requirements:run section of bob.measure:

  run: # this line is already in the recipe. Do not add.
    - python # this line is already in the recipe. Do not add.
    - setuptools # this line is already in the recipe. Do not add.
    - matplotlib
    - boost
    - {{ pin_compatible('numpy') }}
    - docopt

The pin_compatible jinja function is explained in https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#pin-downstream You need to use it on numpy if and only if you use numpy in C level. Otherwise, just list numpy normally. We do not know of any other package besides numpy used in C level that needs to use the pin_compatible jinja function.

Here is a list of packages that we know that they have run_exports:

- bzip2
- dbus
- expat
- ffmpeg
- fontconfig
- freetype
- giflib
- glib
- gmp
- gst-plugins-base
- gstreamer
- hdf5
- icu
- jpeg
- kaldi
- libblitz
- libboost
- libffi
- libmatio
- libogg
- libopus
- libpng
- libsvm
- libtiff
- libvpx
- libxcb
- libxml2
- menpo
- mkl # not this one but mkl-devel has so no need to list mkl if you use mkl-devel in host
- mkl-devel
- ncurses
- openfst
- openssl
- readline
- sox
- speex
- speexdsp
- sqlite
- tk
- vlfeat
- xz
- yaml
- zlib

ENTRY_POINTS_TEST

If you listed some console_sripts in the build:entry_points section, you will test them here. Otherwise, remove the line that says <ENTRY_POINTS_TEST>. For example, if you had the examples entry points above, you would test them like:

test:
  imports:
    - {{ name }}
  commands:
    - jman --help
    - jgen --help

TEST_DEPS

https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#test-requirements You need to list the packages here that are reuired during test-time only. By default, we have bob-devel, nose, coverage, sphinx, ... do not remove them. The test-time dependencies are listed in test-requirements.txt. If this file does not exist, just remove the line that says <TEST_DEPS>.

Left-over conda build files

The conda build command creates a temporary file named record.txt in the project directory. Please make sure it is added in the .gitignore file so that is not committed to the project repository by mistake.

Database packages and packages with extra data

Sometimes databases or other packages require an extra download command after installation. If this extra data is downloaded from Idiap severs, you can include this data in the conda package itself to avoid downloading it two times. If the data is supposed to be downloaded from somewhere other than Idiap servers, do not include it in its conda package. For example, the database packages typically require this download command to be added in the build:script section:

    - python setup.py install --single-version-externally-managed --record record.txt # this line is already in the recipe. Do not add.
    - bob_dbmanage.py {{ name.replace('bob.db.', '') }} download --missing

2. Licensing

Verify if the license of your package satisfies what is written on our spreadsheet:

https://docs.google.com/spreadsheets/d/1jBDLg_VvhDrQYITTepZxn7K69iDIj_FcfRSLk_13AR8/edit#gid=1146180407

Two possible cases:

1. If the package is supposed to be licensed under (a 3-clause) BSD license, then copy and replace the author name in bsd-license.txt. The name of the file should be LICENSE (no extension) and it should sit on the root of your package
2. If the package is supposed to be licensed under GPLv3 license, then copy this file on the root of your package with the name COPYING instead

Important note: change the name and email in setup.py to the maintainer of the package (you in most cases). Andre Anjos is used as an example, do not put his name everywhere.

Download commands (N.B.: replace my name by yours, the package maintainer):

# for 3-clause BSD packages
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/bsd-license.txt > LICENSE
$ sed -i "s/Your Name <your.email@idiap.ch>/Andre Anjos <andre.anjos@idiap.ch>/g;s/<YEAR>/`date +%Y`/g" LICENSE
$ git add LICENSE
$ git rm -f COPYING # if the package used to be GPLv3

# for GPLv3 packages
$ curl -k --silent http://www.gnu.org/licenses/gpl.txt > COPYING
$ git add COPYING
$ git rm -f LICENSE # if the package used to be 3-clause BSD

More info about Idiap's open-source policy here.

Headers

Sometimes people add headers with licensing terms to their files. You should inspect your library to make sure you don't have those. The Idiap TTO says this strategy is OK and simplifies our lives. Make the headers of each file you have as simple as possible, so they don't get outdated in case things change.

Here is a minimal example (adapt to the language comment style if needed):

#!/usr/bin/env python
# vim: set fileencoding=utf-8 :

It is OK to also have your author name on the file if you wish to do so. Don't repeat licensing terms already explained on the root of your package and on the setup.py file. If we need to change the license, it is painful to go through all the headers.

Setup

The setup.py should be changed to:

1. Modify the URL of your package. Make sure it now points to the gitlab page of the package. For example: https://gitlab.idiap.ch/bob/bob.extension
2. Make sure the license is OK. For BSD, it should say license="BSD". For GPLv3, it should say license="GPLv3".
3. Make sure the license is reflected on the classifiers entry. For BSD, it should be 'License :: OSI Approved :: BSD License'. For GPLv3, it should be: 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)'.

$ sed -i "s;github.com/bioidiap;gitlab.idiap.ch/bob;g" setup.py

3. Buildout

The default buildout file buildout.cfg should buildout from the installed distribution and avoid mr.developer checkouts. If you have one of those, move it to develop.cfg and create a new buildout.cfg which should be as simple as possible.

$ git mv buildout.cfg develop.cfg
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/simple-buildout.cfg > buildout.cfg
$ sed -i "s/<DATE>/`date`/g;s/<PACKAGE>/`basename $(pwd)`/g" buildout.cfg
$ sed -i "s/buildout.cfg/buildout.cfg develop.cfg/g" MANIFEST.in
$ sed -i "s;github.com/bioidiap;gitlab.idiap.ch/bob;g" develop.cfg
$ git add buildout.cfg

4. README

Changes are supposed to make the README smaller and easier to maintain. As of today, many packages contain outdated installation instructions or outdated links. More information can always be found at the documentation, which is automatically linked from the badges.

Just copy the template with:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/readme-template.rst > README.rst
$ sed -i "s/<DATE>/`date`/g" README.rst
$ sed -i "s/<PACKAGE>/`basename $(pwd)`/g" README.rst
# example from bob.extension, don't copy verbatim!
$ sed -i "s%<TITLE>%Python/C++ Bob Extension Building Support%g" README.rst
$ sed -i s%<SHORTINTRO>%It provides a simple mechanism for extending Bob_ by building packages using either pure python or a mix of C++ and python.%g;" README.rst

Replace the following tags by hand if you don't like/trust the sed lines above:

1. <DATE>: To today's date. E.g.: Mon 08 Aug 2016 09:47:28 CEST
2. <PACKAGE>: With the name of your package. E.g.: bob.extension
3. <TITLE>: Replace the title (and the size of the title delimiters). E.g.: Python/C++ Bob Extension Building Support
4. <SHORTINTRO>: With a 1 or 2 lines description of your package (it is OK to re-use what you have in setup.py). E.g.: It provides a simple mechanism for extending Bob_ by building packages using either pure python or a mix of C++ and python.

5. Bootstrap

Make sure to remove your bootstrap file from your package. Since a few releases, buildout is not supposed to be bootstrapped anymore as setuptools have dropped support for auto-installing.

Also make sure to remove any mention of bootstrap-buildout.py from the MANIFEST.in file.

6. doc/conf.py

This file needs updating to support newer versions of Sphinx and to get it in sync with the rest of the documentation. Do this:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/sphinx-conf.py > doc/conf.py
$ mkdir -pv doc/img
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/logo.png > doc/img/logo.png
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/favicon.ico > doc/img/favicon.ico
$ sed -i "s/<PROJECT>/`basename $(pwd)`/g" doc/conf.py
# the next line will work if the description in setup.py is correct.
# Otherwise, you need to need to fix "description" in setup.py first.
# Also this will not work with bob.math, for bob.math use this instead:
# sed -i "s%<SHORT_DESCRIPTION>%Mathematical functions of Bob%g" doc/conf.py
$ sed -i "s%<SHORT_DESCRIPTION>%`python setup.py --description`%g" doc/conf.py

The new documentation configuration allows for two optional configuration text files to be placed along conf.py (on the same directory):

• extra-intersphinx.txt, which lists extra packages that should be cross-linked to the documentation (as with Sphinx's intersphinx extension. The format of this text file is simple: it contains the PyPI names of packages to cross-reference. One per line.
• nitpick-exceptions.txt, which lists which documentation objects to ignore (for warnings and errors). The format of this text file is two-column. On the first column, you should refer to Sphinx the object type, e.g. py:class, followed by a space and then the name of the that should be ignored. E.g.: bob.bio.base.Database. The file may optionally contain empty lines. Lines starting with # are ignored (so you can comment on why you're ignoring these objects). Ignoring errors should be used only as a last resource. You should first try to fix the errors as best as you can, so your documentation links are properly working.

7. Update the logo on your project

Please update the logo of your project (on the Settings), just set it to this one.