New Package Instructions

These instructions describe how to migrate or setup your package to gitlab and new licensing terms. At the same time, we profit to update and fix files touched by this procedure. You don't have to do all these steps at once. If you choose to only do some, prioritise by the order in this text (first do the continuous integration changes, then the licensing checks, etc).

Notice: This text may change as we get more experience in what needs to be changed. We may also automatise some of the actions below. We'll keep you posted in this case and update these instructions.

0. Set the right origin

If you haven't checked your repository from gitlab yet, you don't have to remove it and re-clone it, you can just set its origin to point to gitlab:

$ git remote set-url origin git@gitlab.idiap.ch:bob/`basename $(pwd)`
$ git pull

1. Continuous Integration

Remove the .travis.yml file from your package:

$ git rm -f .travis.yml

Copy the appropriate yml template for the CI builds:

• For python-only packages, it should be ci-for-python-only.yml
• For C/C++ extensions, it should be ci-for-cxx-extensions.yml

# for pure python
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-python-only.yml > .gitlab-ci.yml
# for c/c++ extensions
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-cxx-extensions.yml | tr -d '\r' > .gitlab-ci.yml

Add the file to git:

$ git add .gitlab-ci.yml

The ci file should work out of the box. It is long-ish, but generic to any package in the system.

You also need to enable the following options on your project:

1. In the project "Settings" page, make sure builds are enabled
2. If you have a private project, check the package settings and make sure that the "Deploy Keys" for our builders (all conda-* related servers) are enabled
3. Visit the "Runners" section of your package settings and enable all conda runners, for linux and macosx variants
4. Go into the "Variables" section of your package setup and add three variables corresponding to the documentation server, username and password for uploading wheels and latest documentation tar balls to our (web DAV) server. The names of the variables and values are known. If you don't know those, please ask one of us
5. Make sure to enable the service "Build e-mails" and check all optional boxes. You don't need to put any special e-mail addresses
6. Setup the coverage regular expression under "CI/CD pipelines" to have the value ^TOTAL.*\s+(\d+\%)$, which is adequate for figuring out the output of coverage report

2. Licensing

Verify if the license of your package satisfies what is written on our spreadsheet:

https://docs.google.com/spreadsheets/d/1jBDLg_VvhDrQYITTepZxn7K69iDIj_FcfRSLk_13AR8/edit#gid=1146180407

Two possible cases:

1. If the package is supposed to be licensed under (a 3-clause) BSD license, then copy and replace the author name in bsd-license.txt. The name of the file should be LICENSE (no extension) and it should sit on the root of your package
2. If the package is supposed to be licensed under GPLv3 license, then copy this file on the root of your package with the name COPYING instead

Download commands:

# for 3-clause BSD packages
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/bsd-license.txt > LICENSE
$ sed -i "s/Your Name <your.email@idiap.ch>/Andre Anjos <andre.anjos@idiap.ch>/g;s/<YEAR>/`date +%Y`/g" LICENSE
$ git add LICENSE
$ git rm -f COPYING # if the package used to be GPLv3

# for GPLv3 packages
$ curl -k --silent http://www.gnu.org/licenses/gpl.txt > COPYING
$ git add COPYING
$ git rm -f LICENSE # if the package used to be 3-clause BSD

More info about Idiap's open-source policy here.

Headers

Sometimes people add headers with licensing terms to their files. You should inspect your library to make sure you don't have those. The Idiap TTO says this strategy is OK and simplifies our lives. Make the headers of each file you have as simple as possible, so they don't get outdated in case things change.

Here is a minimal example (adapt to the language comment style if needed):

#!/usr/bin/env python
# vim: set fileencoding=utf-8 :

It is OK to also have your author name on the file if you wish to do so. Don't repeat licensing terms already explained on the root of your package and on the setup.py file. If we need to change the license, it is painful to go through all the headers.

Setup

The setup.py should be changed to:

1. Modify the URL of your package. Make sure it now points to the gitlab page of the package. For example: https://gitlab.idiap.ch/bob/bob.extension
2. Make sure the license is OK. For BSD, it should say license="BSD". For GPLv3, it should say license="GPLv3".
3. Make sure the license is reflected on the classifiers entry. For BSD, it should be 'License :: OSI Approved :: BSD License'. For GPLv3, it should be: 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)'.

$ sed -i "s;github.com/bioidiap;gitlab.idiap.ch/bob;g" setup.py

3. Buildout

The default buildout file buildout.cfg should buildout from the installed distribution and avoid mr.developer checkouts. If you have one of those, move it to develop.cfg and create a new buildout.cfg which should be as simple as possible.

$ git mv buildout.cfg develop.cfg
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/simple-buildout.cfg > buildout.cfg
$ sed -i "s/<DATE>/`date`/g;s/<PACKAGE>/`basename $(pwd)`/g" buildout.cfg
$ sed -i "s/buildout.cfg/buildout.cfg develop.cfg/g" MANIFEST.in
$ sed -i "s;github.com/bioidiap;gitlab.idiap.ch/bob;g" develop.cfg
$ git add buildout.cfg

4. README

Changes are supposed to make the README smaller and easier to maintain. As of today, many packages contain outdated installation instructions or outdated links. More information can always be found at the documentation, which is automatically linked from the badges.

Just copy the template with:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/readme-template.rst > README.rst
$ sed -i "s/<DATE>/`date`/g" README.rst
$ sed -i "s/<PACKAGE>/`basename $(pwd)`/g" README.rst
# example from bob.extension, don't copy verbatim!
$ sed -i "s%<TITLE>%Python/C++ Bob Extension Building Support%g" README.rst
$ sed -i s%<SHORTINTRO>%It provides a simple mechanism for extending Bob_ by building packages using either pure python or a mix of C++ and python.%g;" README.rst

Replace the following tags by hand if you don't like/trust the sed lines above:

1. <DATE>: To today's date. E.g.: Mon 08 Aug 2016 09:47:28 CEST
2. <PACKAGE>: With the name of your package. E.g.: bob.extension
3. <TITLE>: Replace the title (and the size of the title delimiters). E.g.: Python/C++ Bob Extension Building Support
4. <SHORTINTRO>: With a 1 or 2 lines description of your package (it is OK to re-use what you have in setup.py). E.g.: It provides a simple mechanism for extending Bob_ by building packages using either pure python or a mix of C++ and python.

5. Bootstrap

Make sure to update your bootstrap file. The latest version is here:

https://bootstrap.pypa.io/bootstrap-buildout.py

Just do a curl to update it:

$ curl -k --silent https://bootstrap.pypa.io/bootstrap-buildout.py > bootstrap-buildout.py

6. doc/conf.py

This file needs updating to support newer versions of Sphinx and to get it in sync with the rest of the documentation. Do this:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/sphinx-conf.py > doc/conf.py
$ mkdir -pv doc/img
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/logo.png > doc/img/logo.png
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/favicon.ico > doc/img/favicon.ico
$ sed -i "s/<PROJECT>/`basename $(pwd)`/g" doc/conf.py
# the next line is **just** an example, change it accordingly to your project
# for example, pick the short_description field in setup.py
$ sed -i "s%<SHORT_DESCRIPTION>%Building of Python/C++ extensions for Bob%g" doc/conf.py

7. Update the logo on your project

Please update the logo of your project (on the Settings), just set it to this one.