Very often, developers are confronted with the need to
Very often, developers are confronted with the need to
clone package repositories locally and develop installation/build and runtime code.
clone package repositories locally and develop installation/build and runtime code.
It is recommended to create isolated environments to develop new projects using conda_ and zc.buildout_.
It is recommended to create isolated environments using conda_ to develop new projects.
Tools implemented in ``bob.devtools`` helps automate this process for |project| packages. In the following we talk about how to checkout and build one or several packages from their git_ source and build proper isolated environments to develop them. Then we will describe how to create a new bob package from scratch and develop existing bob packages along side it.
Tools implemented in ``bob.devtools`` helps automate this process for |project| packages.
In the following we talk about how to checkout and build one or several packages
from their git_ source and build proper isolated environments to develop them.
Then we will describe how to create a new bob package from scratch and develop
existing bob packages along side it.
TLDR
TLDR
====
====
Suppose you want to checkout the package ``bob.blitz`` from source and start developing it locally. We will use the tools implemented in ``bob.devtools`` to create a proper developing environment to build and develop ``bob.blitz``. We assume you have ``bob.devtools`` installed on a conda environment named ``bdt`` (Refer to :ref:`bob.devtools.install` for detailed information.)
Suppose you want to checkout the package ``bob.blitz`` from source and start developing it locally.
We will use the tools implemented in ``bob.devtools`` to create a proper
developing environment to build and develop ``bob.blitz``.
We assume you have ``bob.devtools`` installed on a conda environment named
``bdt`` (Refer to :ref:`bob.devtools.install` for detailed information.)
* Checkout the source of the package from git:
* Checkout the source of the package from git:
.. code-block:: sh
.. code-block:: sh
$ git clone https://gitlab.idiap.ch/bob/bob.blitz
$ bdt dev checkout --use-ssh bob.blitz
* Create a proper conda environment:
* Create a proper conda environment:
.. code-block:: sh
.. code-block:: sh
$ cd bob.blitz
$ cd bob.blitz
$ conda activate bdt
$ bdt dev create -vv dev
$ bdt create -vv dev
$ conda deactivate
$ conda activate dev
$ conda activate dev
* Build the package using buildout:
* Build the package using pip:
.. code-block:: sh
.. code-block:: sh
$ buildout
$ bdt dev install . # calls pip with correct arguments
$ ./bin/python # you should use this python to run things from now on
$ python
for example:
for example:
...
@@ -63,15 +69,7 @@ for example:
...
@@ -63,15 +69,7 @@ for example:
.. code-block:: sh
.. code-block:: sh
$ ./bin/nosetests -sv
$ nosetests -sv
.. note::
Sometimes when you are calling a function not interactively it is not acting normally. In that case import ``pkg_resources`` before importing your package. It is a known issue and we are working on it.
* Some packages may come with a pre-commit_ config file (``.pre-commit-config.yaml``).
* Some packages may come with a pre-commit_ config file (``.pre-commit-config.yaml``).
Make sure to install pre-commit if the config file exists:
Make sure to install pre-commit if the config file exists:
...
@@ -87,18 +85,20 @@ for example:
...
@@ -87,18 +85,20 @@ for example:
Local development of existing packages
Local development of existing packages
======================================
======================================
To develop existing |project| packages you need to checkout their source code and create a proper development environment using `buildout`.
To develop existing |project| packages you need to checkout their source code and install them in your environment.
Checking out package sources
Checking out package sources
----------------------------
----------------------------
|project| packages are developed through gitlab_. Various packages exist
|project| packages are developed through gitlab_. Various packages exist in
in |project|'s gitlab_ instance. In the following we assume you want to install and build locally the ``bob.blitz`` package. In order to checkout a package, just use git_:
|project|'s gitlab_ instance. In the following we assume you want to install and
build locally the ``bob.blitz`` package. In order to checkout a package, just
use git_:
.. code-block:: sh
.. code-block:: sh
$ git clone https://gitlab.idiap.ch/bob/bob.blitz
$ bdt dev checkout --use-ssh bob.blitz
Create an isolated conda environment
Create an isolated conda environment
...
@@ -110,27 +110,30 @@ Before proceeding, you need to make sure that you already have a conda_ environm
...
@@ -110,27 +110,30 @@ Before proceeding, you need to make sure that you already have a conda_ environm
.. code-block:: sh
.. code-block:: sh
$ cd bob.blitz
$ cd bob.blitz
$ conda activate bdt
$ bdt dev create -vv dev
$ bdt create -vv dev
$ conda deactivate
$ conda activate dev
$ conda activate dev
Now you have an isolated conda environment named `dev` with proper channels set. For more information about conda channels refer to `conda channel documentation`_.
Now you have an isolated conda environment named `dev` with proper channels set.
For more information about conda channels refer to `conda channel
documentation`_.
The `bdt create` command assumes a directory named `conda`, exists on the current directory. The `conda` directory contains a file named `meta.yaml`, that is the recipe required to create a development environment for the package you want to develop.
The `bdt dev create` command assumes a directory named `conda`, exists on the
current directory. The `conda` directory contains a file named `meta.yaml`, that
is the recipe required to create a development environment for the package you
want to develop.
.. note::
.. note::
When developing and testing new features, one often wishes to work against the very latest, *bleeding edge*, available set of changes on dependent packages.
When developing and testing new features, one often wishes to work against the very latest, *bleeding edge*, available set of changes on dependent packages.
`bdt create` command adds `Bob beta channels`_ to highest priority which creates an environment with the latest dependencies instead of the latest *stable* versions of each package.
`bdt dev create` command adds `Bob beta channels`_ to highest priority which creates an environment with the latest dependencies instead of the latest *stable* versions of each package.
If you want to create your environment using *stable* channels, you can use this command instead:
If you want to create your environment using *stable* channels, you can use this command instead:
.. code-block:: sh
.. code-block:: sh
$ bdt create --stable -vv dev
$ bdt dev create --stable -vv dev
To see which channels you are using run:
To see which channels you are using run:
...
@@ -147,72 +150,30 @@ The `bdt create` command assumes a directory named `conda`, exists on the curren
...
@@ -147,72 +150,30 @@ The `bdt create` command assumes a directory named `conda`, exists on the curren
environments which can be very different form each other.
environments which can be very different form each other.
Running buildout
Installing the package
----------------
----------------------
The last step is to create a hooked-up environment so you can quickly test
local changes to your package w/o necessarily creating a conda-package.
zc.buildout_ takes care of it by modifying the load paths of scripts to find the correct
version of your package sources from the local checkout. It by default uses a file named `buildout.cfg`, in the package directory. For our example package it looks like:
.. code-block:: ini
; vim: set fileencoding=utf-8 :
The last step is to install the package:
; Mon 08 Aug 2016 14:33:54 CEST
[buildout]
parts = scripts
develop = .
eggs = bob.blitz
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
To find our more information about different section of this file, refer to :ref:`bob.devtools.buildout`.
Now you just need to run buildout:
.. code-block:: sh
.. code-block:: sh
$ cd bob.blitz #if that is not the case
$ cd bob.blitz #if that is not the case
$ conda activate dev #if that is not the case
$ conda activate dev #if that is not the case
$ buildout
$ bdt dev install .
After running, buildout creates a directory called ``bin`` on your local package checkout. Use
the applications living there to develop your package. For example, if you need
to run the test suite:
To run the test suite:
.. code-block:: sh
.. code-block:: sh
$ ./bin/nosetests -sv
$ nosetests -sv
or build the documentation:
or build the documentation:
.. code-block:: sh
.. code-block:: sh
$./bin/sphinx-build -aEn doc sphinx # make sure it finishes without warnings.
$sphinx-build -aEn doc sphinx # make sure it finishes without warnings.
$ firefox sphinx/index.html # view the docs.
$ firefox sphinx/index.html # view the docs.
.. note::
`buildout` by default uses the file `buildout.cfg` but you can specify another file by using -c option. In fact for developing packages especially if they need to be developed along with other packages, another file, namely `develop.cfg` is used like following:
.. code-block:: sh
$ buildout -c develop.cfg
A python interpreter clone can be used to run interactive sessions:
.. code-block:: sh
$ ./bin/python
You can see what is installed in your environment:
You can see what is installed in your environment:
...
@@ -233,54 +194,29 @@ And you can install new packages using conda:
...
@@ -233,54 +194,29 @@ And you can install new packages using conda:
.. code-block:: sh
.. code-block:: sh
$ cd <package>
$ cd <package>
$ conda activate bdt
$ bdt local build
$ bdt local build
One important advantage of using conda_ and zc.buildout_ is that it does
One important advantage of using conda_ is that it does **not** require
**not** require administrator privileges for setting up any of the above.
administrator privileges for setting up any of the above. Furthermore, you will
Furthermore, you will be able to create distributable environments for each
be able to create distributable environments for each project you have. This is
project you have. This is a great way to release code for laboratory exercises
a great way to release code for laboratory exercises or for a particular
or for a particular publication that depends on |project|.
It so happens that you want to develop several packages against each other for your project. Let's assume you want to develop ``bob.blitz`` and ``bob.extension`` simultaneously. ``bob.blitz`` is dependent on ``bob.devtools``. First we checkout package ``bob.blitz`` and build an isolated conda environment as explained in the previous section. Then edit `buildout.cfg` file (or `develop.cfg`) and add ``bob.extension`` to it as following:
It so happens that you want to develop several packages against each other for
your project. Let's assume you want to develop ``bob.blitz`` and
``bob.extension`` simultaneously. ``bob.blitz`` is dependent on
.. code-block:: ini
``bob.extension``. First we checkout package ``bob.blitz`` and build an isolated
conda environment as explained in the previous section. Then checkout and
Now you can run `buildout` as usual. The ``bob.extension`` will be checked out on `src` folder on the root of your project.
.. note::
The flag `debug = true` is usually used when in development mode.
$ bdt dev checkout --use-ssh --subfolder src bob.extension
$ bdt dev install src/bob.extension
.. _bob.devtools.create_package:
.. _bob.devtools.create_package:
...
@@ -288,98 +224,56 @@ Now you can run `buildout` as usual. The ``bob.extension`` will be checked out o
...
@@ -288,98 +224,56 @@ Now you can run `buildout` as usual. The ``bob.extension`` will be checked out o
Local development of a new package
Local development of a new package
==================================
==================================
In this section we explain how to create a new bob package from scratch and start developing it. Once again ``bob.devtools`` is here to help you. You need to activate your conda environment with ``bob.devtools`` installed in it.
In this section we explain how to create a new bob package from scratch and
start developing it. Once again ``bob.devtools`` is here to help you. You need
to activate your conda environment with ``bob.devtools`` installed in it.
.. code-block:: sh
.. code-block:: sh
$ conda activate bdt
$ bdt new -vv bob/bob.project.awesome author_name author_email
$ bdt new -vv bob/bob.project.awesome author_name author_email
This command will create a new bob package named "bob.project.awesome" that includes the correct anatomy of a package. For more information about the functionality of each file check :ref:`bob.devtools.anatomy`.
This command will create a new bob package named "bob.project.awesome" that
includes the correct anatomy of a package. For more information about the
In the root of your project there is a file `buildout.cfg` used by `buildout` to build your package locally. It should look like:
functionality of each file check :ref:`bob.devtools.anatomy`.
.. code-block:: ini
[buildout]
parts = scripts
develop = .
eggs = bob.project.awesome
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true
Now you have all the necessary tools available and you can make a development environment using `bdt create` command, run `buildout` in it and start developing your package.
Now you have all the necessary tools available and you can make a development
environment using `bdt dev create` command, run `bdt dev install` in it and
start developing your package.
.. code-block:: sh
.. code-block:: sh
$ cd bob.project.awesome
$ cd bob.project.awesome
$ conda activate bdt
$ bdt dev create --stable -vv awesome-project #here we used the stable channels to make the conda environment.
$ bdt create --stable -vv awesome-project #here we used the stable channels to make the conda environment.
$ conda activate awesome-project
$ conda activate awesome-project
$ buildout
$ bdt dev install .
Developing existing bob packages along with your new package
Developing existing bob packages along with your new package
$ bdt dev checkout --use-ssh --subfolder src bob.extension bob.blitz
$ bdt dev install src/bob.extension src/bob.blitz # the order of installing dependencies matter!
When you build your new package the dependent packages (in this example ``bob.extension`` and ``bob.blitz``) will be checked out on folder `src` in the root of your project.
When you build your new package, it is customary to checkout the dependent
packages (in this example ``bob.extension`` and ``bob.blitz``) in the `src`
folder in the root of your project.
As usual, first create an isolated conda environment using `bdt create` command. Some of bob packages need dependencies that might not be installed on your environment. You can find these dependencies by checking `conda/meta.yaml` of each package. Install the required packages and then run buildout as usual. For our example you need to do the following:
As usual, first create an isolated conda environment using `bdt dev create` command.
Some of bob packages need dependencies that might not be installed on your
environment. You can find these dependencies by checking `conda/meta.yaml` of
each package. Install the required packages and then run `bdt dev install`. For
$ bdt dev install src/bob.extension src/bob.blitz # the order of installing dependencies matter!
.. note::
Sometimes you may need some of bob packages available in your local `bin` directory without necessarily developing them.
If you knew beforehand what are those packages, you can add them to "requirements/host" section of the `conda/meta.yaml` file and then create a conda environment using `bdt create`. Like this, those packages will be installed automatically. Otherwise, if you already have your conda environment, install them using `conda install` command.
When done, add those packages to the `eggs` section in your `buildout.cfg` file and then run `buildout`.
