.. * Install dependencies for `bob.extension` and `bob.blitz` using `conda install` command. You can find out the required dependencies by looking at `conda/meta.yaml` file for each package.
Sometimes when you are calling a function not interactively it is not acting normally. In that case import ``pkg_resources`` before importing your package. It is a known issue and we are working on it.
Sometimes when you are calling a function not interactively it is not acting normally. In that case import ``pkg_resources`` before importing your package. It is a known issue and we are working on it.
...
@@ -253,30 +244,30 @@ It so happens that you want to develop several packages against each other for y
...
@@ -253,30 +244,30 @@ It so happens that you want to develop several packages against each other for y
Now you can run `buildout` as usual. The `bob.<package-B>` will be checked out on `src` folder on the root of your project.
Now you can run `buildout` as usual. The ``bob.extension`` will be checked out on `src` folder on the root of your project.
.. note::
.. note::
...
@@ -285,42 +276,17 @@ Now you can run `buildout` as usual. The `bob.<package-B>` will be checked out o
...
@@ -285,42 +276,17 @@ Now you can run `buildout` as usual. The `bob.<package-B>` will be checked out o
.. _bob.devtools.create_package:
.. _bob.devtools.create_package:
Local development of a new Bob package
Local development of a new package
======================================
==================================
In this section we explain how to create a new bob package from scratch and start developing it. Once again `bob.devtools` is here to help you. You need to activate your conda environment with bob.devtools installed in it.
In this section we explain how to create a new bob package from scratch and start developing it. Once again ``bob.devtools`` is here to help you. You need to activate your conda environment with ``bob.devtools`` installed in it.
.. code-block:: sh
.. code-block:: sh
$ conda activate bdt
$ conda activate bdt
$ bdt new -vv bob/bob.project.awesome author_name author_email
$ bdt new -vv bob/bob.project.awesome author_name author_email
This command will create a new bob package named "awesome-project" that includes the correct anatomy of a package. Here we briefly explain the purpose of some of the parts:
This command will create a new bob package named "awesome-project" that includes the correct anatomy of a package. For more information about the functionality of each file check :ref:`bob.devtools.anatomy`.
.. code-block:: text
bob.<awesome-project>
+-- bob
+-- __init__.py #namespace init for "bob"
+-- conda
+-- meta.yaml
+-- doc
+-- img
+-- conf.py
+-- index.rst
+-- links.rst
+-- .gitignore
+-- .gitlab-ci.yml
+-- buildout.cfg
+-- COPYING
+-- MANIFEST.IN
+-- README.rst
+-- requirements.txt
+-- setup.py
+-- version.txt
For more information about the functionality of each file check :ref:`bob.devtools.anatomy`.
In the root of your project there is a file `buildout.cfg` used by `buildout` to build your package locally. It should look like:
In the root of your project there is a file `buildout.cfg` used by `buildout` to build your package locally. It should look like:
...
@@ -345,7 +311,7 @@ Now you have all the necessary tools available and you can make a development en
...
@@ -345,7 +311,7 @@ Now you have all the necessary tools available and you can make a development en
$ cd bob.project.awesome
$ cd bob.project.awesome
$ conda activate bdt
$ conda activate bdt
$ bdt create --stable -vv awesome-project
$ bdt create --stable -vv awesome-project #here we used the stable channels to make the conda environment.
$ conda activate awesome-project
$ conda activate awesome-project
$ buildout
$ buildout
...
@@ -353,9 +319,9 @@ Now you have all the necessary tools available and you can make a development en
...
@@ -353,9 +319,9 @@ Now you have all the necessary tools available and you can make a development en
Developing existing bob packages along with your new package
Developing existing bob packages along with your new package
When you build your new package the dependent packages (in this example `bob.extension` and `bob.blitz`) will be checked out on folder `src` in the root of your project.
When you build your new package the dependent packages (in this example ``bob.extension`` and ``bob.blitz``) will be checked out on folder `src` in the root of your project.
As usual, first create an isolated conda environment using `bdt create` command. Some of bob packages need dependencies that might not be installed on your environment. You can find these dependencies by checking `conda/meta.yaml` of each package. Install the required packages and then run buildout as usual.
As usual, first create an isolated conda environment using `bdt create` command. Some of bob packages need dependencies that might not be installed on your environment. You can find these dependencies by checking `conda/meta.yaml` of each package. Install the required packages and then run buildout as usual. For our example you need to do the following:
Sometimes you may need some of bob packages available in your local `bin` directory without necessarily developing them.
Sometimes you may need some of bob packages available in your local `bin` directory without necessarily developing them.
If you knew beforehand what are those packages, you can add them to "requirements/host" section of the `conda/meta.yaml` file and then create a conda environment using `bdt create`. Like this those packages will be installed automatically. Otherwise, if you already have your conda environment, install them using `conda install` command.
If you knew beforehand what are those packages, you can add them to "requirements/host" section of the `conda/meta.yaml` file and then create a conda environment using `bdt create`. Like this, those packages will be installed automatically. Otherwise, if you already have your conda environment, install them using `conda install` command.
When done, add those packages to the `eggs` section in your `buildout.cfg` file and then run `buildout`.
When done, add those packages to the `eggs` section in your `buildout.cfg` file and then run `buildout`.
.. _bob.devtools.buildout:
Some notes on buildout
======================
To be able to develop a package, we first need to build and install it locally.
While developing a package, you need to install your package in *development*
mode so that you do not have to re-install your package after every change that
you do in the source. zc.buildout_ allows you to exactly do that.
.. note::
zc.buildout_ will create another local environment from your conda_
environment but unlike conda_ environments this environment is not isolated
rather it inherits from your conda_ environment. This means you can still
use the libraries that are installed in your conda_ environment.
zc.buildout_ also allows you to install PyPI_ packages into your
environment. You can use it to install some Python library if it is not
available using conda_. Keep in mind that to install a library you should
always prefer conda_ but to install your package from source in
*development* mode, you should use zc.buildout_.
zc.buildout_ provides a ``buildout`` command. ``buildout`` takes as input a
"recipe" that explains how to build a local working environment. The recipe, by
default, is stored in a file called ``buildout.cfg``.
.. note::
Buildout by default looks for ``buildout.cfg`` in your current folder and
uses that configuration file. You can specify a different config file with
the ``-c`` option:
.. code:: sh
$ buildout -c develop.cfg
.. important::
Once ``buildout`` runs, it creates several executable scripts in a local
``bin`` folder. Each executable is programmed to use Python from the conda
environment, but also to consider (prioritarily) your package checkout.
This means that you need to use the scripts from the ``bin`` folder instead
of using its equivalence from your conda environment. For example, use
``./bin/python`` instead of ``python``.
``buildout`` will examine the ``setup.py`` file of packages using setuptools_
and will ensure all build and run-time dependencies of packages are available
either through the conda installation or it will install them locally without
changing your conda environment.
The configuration file is organized in several *sections*, which are indicated
by ``[]``, where the default section ``[buildout]`` is always required. Some of
the entries need attention.
* The first entry are the ``eggs``. In there, you can list all python packages
that should be installed. These packages will then be available to be used in
your environment. Dependencies for those packages will be automatically
managed, **as long as you keep** ``bob.buildout`` **in your list of**
``extensions``. At least, the current package needs to be in the ``eggs``
list.
* The ``extensions`` list includes all extensions that are required in the
buildout process. By default, only ``bob.buildout`` is required, but more
extensions can be added (more on that later).
* The next entry is the ``develop`` list. These packages will be installed
*development mode* from the specified folder.
The remaining options define how the (dependent) packages are built. For
example, the ``debug`` flag defined, how the *C++ code* in
all the (dependent) packages is built. For more information refer to *C/C++ modules in your package* in `bob.extension <https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/index.html>`_ documentation. The ``verbose`` options handles the
verbosity of the build. When the ``newest`` flag is set to ``true``, buildout
will install all packages in the latest versions, even if an older version is
already available.
.. note::
We normally set ``newest = False`` to avoid downloading already installed
dependencies. Also, it installs by default the latest stable version of the
package, unless ``prefer-final = False``, in which case the latest
available on PyPI, including betas, will be installed.
.. warning::
Compiling packages in debug mode (``debug = true``) will make them very
slow. You should only use this option when you are developing and not for
running experiments or production.
When the buildout command is invoked it will perform the following steps:
1. It goes through the list of ``eggs``, searched for according packages and
installed them *locally*.
2. It populates the ``./bin`` directory with all the ``console_scripts`` that
you have specified in the ``setup.py``.
.. important::
One thing to note in package development is that when you change the entry
points in ``setup.py`` of a package, you need to run ``buildout`` again.
.. _bob.devtools.mr.developer:
Using mr.developer
==================
One extension that may be useful is `mr.developer`_. It allows to develop
*several packages* at the same time. This extension will allow
buildout to automatically check out packages from git repositories, and places
them into the ``./src`` directory. It can be simply set up by adding
``mr.developer`` to the extensions section.
In this case, the develop section should be augmented with the packages you
would like to develop. There, you can list directories that contain Python
packages, which will be build in exactly the order that you specified. With
this option, you can tell buildout particularly, in which directories it should
@@ -147,6 +147,222 @@ Buildout.cfg in more details
...
@@ -147,6 +147,222 @@ Buildout.cfg in more details
This section should include more information about different sections in a buildout.cfg file.
This section should include more information about different sections in a buildout.cfg file.
Some notes on buildout
======================
To be able to develop a package, we first need to build and install it locally.
While developing a package, you need to install your package in *development*
mode so that you do not have to re-install your package after every change that
you do in the source. zc.buildout_ allows you to exactly do that.
.. note::
zc.buildout_ will create another local environment from your conda_
environment but unlike conda_ environments this environment is not isolated
rather it inherits from your conda_ environment. This means you can still
use the libraries that are installed in your conda_ environment.
zc.buildout_ also allows you to install PyPI_ packages into your
environment. You can use it to install some Python library if it is not
available using conda_. Keep in mind that to install a library you should
always prefer conda_ but to install your package from source in
*development* mode, you should use zc.buildout_.
zc.buildout_ provides a ``buildout`` command. ``buildout`` takes as input a
"recipe" that explains how to build a local working environment. The recipe, by
default, is stored in a file called ``buildout.cfg``.
.. note::
Buildout by default looks for ``buildout.cfg`` in your current folder and
uses that configuration file. You can specify a different config file with
the ``-c`` option:
.. code:: sh
$ buildout -c develop.cfg
.. important::
Once ``buildout`` runs, it creates several executable scripts in a local
``bin`` folder. Each executable is programmed to use Python from the conda
environment, but also to consider (prioritarily) your package checkout.
This means that you need to use the scripts from the ``bin`` folder instead
of using its equivalence from your conda environment. For example, use
``./bin/python`` instead of ``python``.
``buildout`` will examine the ``setup.py`` file of packages using setuptools_
and will ensure all build and run-time dependencies of packages are available
either through the conda installation or it will install them locally without
changing your conda environment.
The configuration file is organized in several *sections*, which are indicated
by ``[]``, where the default section ``[buildout]`` is always required. Some of
the entries need attention.
* The first entry are the ``eggs``. In there, you can list all python packages
that should be installed. These packages will then be available to be used in
your environment. Dependencies for those packages will be automatically
managed, **as long as you keep** ``bob.buildout`` **in your list of**
``extensions``. At least, the current package needs to be in the ``eggs``
list.
* The ``extensions`` list includes all extensions that are required in the
buildout process. By default, only ``bob.buildout`` is required, but more
extensions can be added (more on that later).
* The next entry is the ``develop`` list. These packages will be installed
*development mode* from the specified folder.
The remaining options define how the (dependent) packages are built. For
example, the ``debug`` flag defined, how the *C++ code* in
all the (dependent) packages is built. For more information refer to *C/C++ modules in your package* in `bob.extension <https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/index.html>`_ documentation. The ``verbose`` options handles the
verbosity of the build. When the ``newest`` flag is set to ``true``, buildout
will install all packages in the latest versions, even if an older version is
already available.
.. note::
We normally set ``newest = False`` to avoid downloading already installed
dependencies. Also, it installs by default the latest stable version of the
package, unless ``prefer-final = False``, in which case the latest
available on PyPI, including betas, will be installed.
.. warning::
Compiling packages in debug mode (``debug = true``) will make them very
slow. You should only use this option when you are developing and not for
running experiments or production.
When the buildout command is invoked it will perform the following steps:
1. It goes through the list of ``eggs``, searched for according packages and
installed them *locally*.
2. It populates the ``./bin`` directory with all the ``console_scripts`` that
you have specified in the ``setup.py``.
.. important::
One thing to note in package development is that when you change the entry
points in ``setup.py`` of a package, you need to run ``buildout`` again.
.. _bob.devtools.mr.developer:
Using mr.developer
==================
One extension that may be useful is `mr.developer`_. It allows to develop
*several packages* at the same time. This extension will allow
buildout to automatically check out packages from git repositories, and places
them into the ``./src`` directory. It can be simply set up by adding
``mr.developer`` to the extensions section.
In this case, the develop section should be augmented with the packages you
would like to develop. There, you can list directories that contain Python
packages, which will be build in exactly the order that you specified. With
this option, you can tell buildout particularly, in which directories it should
Everything is now setup for you to continue the development of the packages.
Moreover, you can learn more about |project| packages and learn to create new
ones in .
.. _bob.devtools.anatomy:
.. _bob.devtools.anatomy:
...
@@ -154,6 +370,30 @@ Buildout.cfg in more details
...
@@ -154,6 +370,30 @@ Buildout.cfg in more details
Anatomy of a new package
Anatomy of a new package
========================
========================
.. code-block:: text
bob.<awesome-project>
+-- bob
+-- __init__.py #namespace init for "bob"
+-- conda
+-- meta.yaml
+-- doc
+-- img
+-- conf.py
+-- index.rst
+-- links.rst
+-- .gitignore
+-- .gitlab-ci.yml
+-- buildout.cfg
+-- COPYING
+-- MANIFEST.IN
+-- README.rst
+-- requirements.txt
+-- setup.py
+-- version.txt
There is a folder named `conda` that includes a file `meta.yaml`. As explained earlier this files includes the information used to prepare a proper conda environment.
There is a folder named `conda` that includes a file `meta.yaml`. As explained earlier this files includes the information used to prepare a proper conda environment.
The folder named `bob` which should only include a file `__init__.py` at this stage is where you will put all your new code and functionality in.
The folder named `bob` which should only include a file `__init__.py` at this stage is where you will put all your new code and functionality in.
...
@@ -163,6 +403,10 @@ The folder named `doc` includes the minimum necessary information for building p
...
@@ -163,6 +403,10 @@ The folder named `doc` includes the minimum necessary information for building p
`.gitlab-ci.yml` includes the information about building packages on the ci. We will talk about it later.
`.gitlab-ci.yml` includes the information about building packages on the ci. We will talk about it later.
