Skip to content
Snippets Groups Projects
Commit 26aca3c8 authored by André Anjos's avatar André Anjos :speech_balloon:
Browse files

Merge branch 'issue-25' into 'master' 

Issue 25

Closes #25

See merge request !29
parents 71e6ff30 4b5354f3
No related branches found
No related tags found
1 merge request!29Issue 25
Pipeline #
Hide whitespace changes
Inline Side-by-side
README.rst
+ 13
− 151
View file @ 26aca3c8
.. vim: set fileencoding=utf-8 : .. vim: set fileencoding=utf-8 :
.. Thu 30 Jan 08:46:53 2014 CET .. Tue 27 Feb 2018 09:06:28 CET
.. image:: http://img.shields.io/badge/docs-stable-yellow.png .. image:: https://img.shields.io/badge/docs-stable-yellow.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/stable/index.html :target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/stable/index.html
.. image:: http://img.shields.io/badge/docs-latest-orange.png .. image:: https://img.shields.io/badge/docs-latest-orange.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/master/index.html :target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/master/index.html
.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/build.svg .. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/build.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master :target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/coverage.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
.. image:: https://img.shields.io/badge/gitlab-project-0000c0.svg .. image:: https://img.shields.io/badge/gitlab-project-0000c0.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout :target: https://gitlab.idiap.ch/bob/bob.buildout
.. image:: http://img.shields.io/pypi/v/bob.buildout.png .. image:: https://img.shields.io/pypi/v/bob.buildout.svg
:target: https://pypi.python.org/pypi/bob.buildout :target: https://pypi.python.org/pypi/bob.buildout
...@@ -18,157 +20,17 @@ ...@@ -18,157 +20,17 @@
=================================== ===================================
This package is part of the signal-processing and machine learning toolbox This package is part of the signal-processing and machine learning toolbox
Bob_. It contains a number of ``zc.buildout`` recipes for simplifying builds of Bob_. It contains a number of ``zc.buildout`` recipes for simplifying
Bob_ packages. development of Bob_ packages.
C++/Python Extension
--------------------
This extension allows you to compile C/C++ extensions that depend on each other
using a buildout. It assures that eggs living in both ``develop-eggs`` and
``eggs`` are on your path before building the packages in the ``develop``
section. By using this extension you can drop the use of the local recipe
``bob.buildout:develop``, which should be considered deprecated.
Supported Options
=================
verbose
If set, buildout it will output the compilation commands while compiling the
module.
prefixes
A list of directories where this recipe will look for installed software,
such as compiled libraries and header files. It is the same as setting the
environment variable ``BOB_PREFIX_PATH`` to a list of paths containing
externally installed software. As a side-effect, setting ``BOB_PREFIX_PATH``
also sets, internally, ``PKG_CONFIG_PATH`` to a list of directories following
where to search for pkg-config files.
debug
If set, the module will be compiled with debugging symbols and with
optimization turned off. If ``debug`` is set to ``true``, this is equivalent
to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
-g``. If it is set to ``false``, it is the same as appending ``-O3 -g0``
instead.
environ
The name of a section on your configuration file that contains the names and
values of environment variables that should be used through the build. This
section is named, by default, ``environ``.
If a section named ``environ`` exists, it is read and the environment
variables are set **before** the specified eggs are developed. You can use
variable substitution on this section. Here is an an example::
[environ]
CFLAGS = '-O0 -g -DNDEBUG'
CXXFLAGS = ${CFLAGS}
Notice there is some functionality overlap between the previous flags and the
use of section ``environ``. While it is more flexible, you must understand
the consequences of setting both ``prefixes`` and ``debug``, together with
``environ``. The rule is simple: values set on the ``environ`` section have
**precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
to those listed in ``prefixes``, if that is also set.
Multi-Script Installer
----------------------
This recipe installs **all** most used scripts and interpreter proxies for your
package. It will look at the ``buildout`` section entry called ``prefixes``,
that potentially lists prefixes that should be **prepended** to the default
python environment. In these prefixes, it will look for standard python
directories. If one or more are found, these paths are **prepended** into
the resulting scripts generated by this recipe and eggs will be searched on
those locations prioritarily.
By default, this recipe will use the eggs defined at the ``buildout`` section
called ``eggs``, but that can be overriden locally. It generates these scripts:
python
A pre-configured python interpreter
gdb-python
A pre-configured python interpreter, prefixed with ``gdb`` to make debugging
easier. Use it like you use ``python``.
nosetests
A test runner called ``nosetests`` will be created on the bin directory of
buildout.
coverage
A test coverage application called ``coverage`` will be created on the bin
directory of buildout.
sphinx
Several sphinx utilities will be created on the bin directory of buildout.
package scripts
Package scripts will be created taking into account the ``prefixes``
established for this section or globally (as a second priority).
To use this recipe, you just have to simply do::
[scripts]
recipe = bob.buildout:scripts
Common Supported Options
========================
The recipe supports the following options:
prefixes
A list of directories where this recipe will look for subdirectories with
the stem ``lib/python*/site-packages``. All directories matching this
condition are appended to the search paths. If not given, the value of this
property defaults to ``buildout.prefixes``. Both can be empty, which makes
this recipe default to using standard available paths.
eggs
The eggs option specifies a list of eggs to use for **building** this
package. Each string must be given on a separate line. If not given, the
value of this property defaults to ``buildout.eggs``.
dependent-scripts
If set to the string ``true``, scripts will be generated for all required
eggs in addition to the eggs specifically named.
interpreter
The name of a script to generate that allows access to a Python interpreter
that has the path set based on the eggs installed. If you don't specify
anything, the default value ``python`` will be used.
extra-paths
Extra paths to be appended in a generated script. To prepend, using the
``prefixes`` entry.
nose-flags
These are extra flags that are **appended** to the given ``nosetests``
command line, automatically. Use this to preset arguments you like running
all the time like ``-v``, for example.
Installation Installation
------------ ------------
Follow our `installation`_ instructions. Then, using the Python interpreter Complete Bob's `installation`_ instructions. Then, to install this package,
provided by the distribution, bootstrap and buildout this package:: run::
$ python bootstrap-buildout.py $ conda install bob.buildout
$ ./bin/buildout
Contact Contact
...@@ -180,5 +42,5 @@ development `mailing list`_. ...@@ -180,5 +42,5 @@ development `mailing list`_.
.. Place your references here: .. Place your references here:
.. _bob: https://www.idiap.ch/software/bob .. _bob: https://www.idiap.ch/software/bob
.. _installation: https://gitlab.idiap.ch/bob/bob/wikis/Installation .. _installation: https://www.idiap.ch/software/bob/install
.. _mailing list: https://groups.google.com/forum/?fromgroups#!forum/bob-devel .. _mailing list: https://www.idiap.ch/software/bob/discuss
doc/buildout.rst 0 → 100644
+ 224
− 0
View file @ 26aca3c8
.. bob.buildout.recipes:
=======================
Extension and Recipes
=======================
This package is composed of an extension to zc.buildout_ and a recipe. We
recommend you use both during the creation of your own package-based recipes.
Here is a typical ``buildout.cfg`` file that can be used to setup the
development of your own package, and which we normally ship with all
|project|-based packages:
.. code-block:: ini
[buildout]
parts = scripts
develop = .
eggs = <PACKAGE>
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
Replace ``<PACKAGE>`` by your package name and you should be ready to run the
``buildout`` application. If you're curious about zc.buildout_ and how to
structure your recipe to take full advantage of it, we advise you seek
documentation on that package on its website or by searching the internet.
The above setup will include all required material for building both simple
python or python/C/C++ packages from |project|.
By using extension ``bob.buildout`` on the ``buildout`` section of your recipe,
you ensure the current package will be built taking into consideration all
required environment/compiler settings required by |project|. This extension is
strictly required for packages containing C/C++ bindings, but it is harmless to
include it in Python-only packages.
The section ``scripts`` define a list of scripts that will be installed on the
``bin`` directory. By default, we make sure all packages listed in
``buildout.eggs`` are available, as well as typical packages required for
development such as ``sphinx``, for documentation building, ``nose`` for
running your test suite and ``coverage``, for code coverage reporting. You may
augment the ``eggs`` entry on the ``buildout`` section if you'd like further
packages to be installed on your development environment. It is your call. An
example package that is often listed there is ``ipdb`` - the iPython based
python debugger. The syntax of the ``eggs`` entry on the ``buildout`` section
is one package per line.
The entry ``develop`` on the ``buildout`` section indicates to buildout the
root directories of packages that it should take, prioritarily, before
considering potential versions installed on base python (conda) environment. It
typically says just ``.`` (dot), as we're typically willing to *only* make
buildout aware of our local checkout. It can include more directories (one per
line), if we'd like to cross-develop more packages. That is what we do, for
example, when using the extension ``mr.developer`` (see:
:ref:`bob.extension` development). For each package listed in ``develop``,
buildout will run the equivalent of ``cd <dir> && python setup.py develop`` on
each package **before** trying to resolve the dependencies in ``eggs``. It will
then consider those locally installed packages in final load-path scheme for
your applications.
In case you're curious, the ``buildout.newest`` flag is internal to
zc.buildout_. It instructs the setuptools_ machinery to consider or not
versions of packages currently installed. If ``newest=true``, then only the
absolute newest packages in PyPI_ can satisfy the build. By default, we let
this setting to ``false`` indicating that what is currently installed will be
sufficient. Only change it if you know the implications. Note, for example, in
this case, you may not be testing anymore, exclusively, against a
|project|-agreed set of dependencies.
The remainder flags and options are explained next.
Supported extension options
---------------------------
Here is a list of supported options for the ``bob.buildout``. They should
appear in the ``buildout`` section of your recipe. They are considered for as
long as ``bob.buildout`` is listed as an extension.
``verbose``
If set, buildout it will output the compilation commands while compiling the
module. This flag should go into the ``buildout`` section of your recipe.
Accepted values are ``true`` or ``false``.(the default).
``prefixes``
A list of directories where this recipe will look for installed software,
such as compiled libraries and header files. It is important if you're
compiling a package that requires external software such as ffmpeg or blitz++
header files to be available. It is the same as setting the environment
variable ``BOB_PREFIX_PATH`` to a list of paths containing externally
installed software. As a side-effect, setting ``BOB_PREFIX_PATH`` also sets,
internally, ``PKG_CONFIG_PATH`` to a list of directories following where to
search for pkg-config files.
You typically don't need to set this manually since ``bob.extension`` is
smart enough to figure out where to find external libraries. This is even
more true if you're using conda-based installations as the bootstrapping
environment of your buildout.
``debug``
If set, the module will be compiled with debugging symbols and with
optimization turned off. If ``debug`` is set to ``true``, this is equivalent
to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
-g``. By default, the value is ``false``. This setting is advised if you are
compiling python C/C++ bindings and would like to debug C/C++ code locally.
``environ``
The name of another section on your configuration file that contains the
names and values of environment variables that should be set before any
possible build takes place. This section is named, by default, ``environ``.
That is, if you provide a section named ``environ`` on your buildout recipe,
then you don't need to explicitly specify its name using this flag.
If a section named ``environ`` (or whatever lists the ``environ`` override on
the ``buildout`` section of your recipe) exists, it is read and the
environment variables are set **before** the specified packages are built.
You can use variable substitution on this section. Here is an an example::
[environ]
CFLAGS = '-O0 -g -DNDEBUG'
CXXFLAGS = ${CFLAGS}
Notice there is some functionality overlap between the previous flags and the
use of section ``environ``. While it is more flexible, you must understand
the consequences of setting both ``prefixes`` and ``debug``, together with
``environ``. The rule is simple: values set on the ``environ`` section have
**precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
to those listed in ``prefixes``, if that is also set.
The ``scripts`` recipe
----------------------
By using the recipe ``bob.buildout:scripts`` on one of the sections of your
recipe, you ensure the scripts generated by the recipe will be built taking
into consideration all installed packages from your base python environment
(typically a conda-based installation). If you don't use the
``bob.buildout:scripts`` recipe, zc.buildout_, by default, assumes no packages
are availabe on the python installation and may download/recompile all
dependencies from scratch.
By default, this recipe will use the eggs defined at the ``buildout`` section
called ``eggs``, but that can be overriden locally. It generates these scripts:
``python``
A pre-configured python interpreter taking into consideration all eggs and
development packages on your recipe.
``gdb-python`` or ``lldb-python``
A pre-configured python interpreter, prefixed with ``gdb`` (or ``lldb`` on a
MacOS system) to make debugging easier. Use it like you use ``python``.
``nosetests``
A test runner called ``nosetests`` will be created on the bin directory of
buildout.
``coverage``
A test coverage application called ``coverage`` will be created on the
``bin`` directory of buildout.
``sphinx-build`` and friends
Several sphinx utilities will be created on the bin directory of buildout.
Other package scripts
Package scripts will be created taking into account the ``prefixes``
established for this section or globally (as a second priority).
Supported recipe options
========================
The ``scripts`` recipe supports the following options (mostly for experts -
read don't use them unless you know what you're doing):
``prefixes``
Overrides or sets the list of prefixes in ``buildout.prefixes``. If not
given, the value of this property defaults to ``buildout.prefixes``. Both can
be empty, which makes this recipe default to using standard available paths.
``eggs``
The eggs option specifies a list of eggs to use for **building** the scripts
of this recipe. Each string must be given on a separate line. If not given,
the value of this property defaults to ``buildout.eggs``.
``dependent-scripts``
If set to the string ``true``, scripts will be generated for all required
eggs in addition to the eggs specifically named. By default, it is ``false``
to avoid the clutter it may cause on very high-level packages, with numerous
dependencies exporting scripts.
``interpreter``
The name of a script to generate that allows access to a Python interpreter
that has the path set based on the eggs installed. If you don't specify
anything, the default value ``python`` will be used.
``extra-paths``
Extra paths to be appended in a generated script. To prepend, use the
``prefixes`` entry.
``nose-flags``
These are extra flags that are **appended** to the given ``nosetests``
command line, automatically. Use this to preset arguments you like running
all the time like ``-v``, for example.
.. include:: links.rst
doc/conf.py
+ 61
− 3
View file @ 26aca3c8
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
import os import os
import sys import sys
import glob import glob
import pkg_resources
# -- General configuration ----------------------------------------------------- # -- General configuration -----------------------------------------------------
...@@ -27,6 +28,25 @@ extensions = [ ...@@ -27,6 +28,25 @@ extensions = [
'sphinx.ext.mathjax', 'sphinx.ext.mathjax',
] ]
# Be picky about warnings
nitpicky = True
# Ignores stuff we can't easily resolve on other project's sphinx manuals
nitpick_ignore = []
# Allows the user to override warnings from a separate file
if os.path.exists('nitpick-exceptions.txt'):
for line in open('nitpick-exceptions.txt'):
if line.strip() == "" or line.startswith("#"):
continue
dtype, target = line.split(None, 1)
target = target.strip()
try: # python 2.x
target = unicode(target)
except NameError:
pass
nitpick_ignore.append((dtype, target))
# Always includes todos # Always includes todos
todo_include_todos = True todo_include_todos = True
...@@ -57,14 +77,17 @@ project = u'bob.buildout' ...@@ -57,14 +77,17 @@ project = u'bob.buildout'
import time import time
copyright = u'%s, Idiap Research Institute' % time.strftime('%Y') copyright = u'%s, Idiap Research Institute' % time.strftime('%Y')
# Grab the setup entry
distribution = pkg_resources.require(project)[0]
# The version info for the project you're documenting, acts as replacement for # The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the # |version| and |release|, also used in various other places throughout the
# built documents. # built documents.
# #
# The short X.Y version. # The short X.Y version.
version = open('../version.txt').read().strip() version = distribution.version
# The full version, including alpha/beta/rc tags. # The full version, including alpha/beta/rc tags.
release = version release = distribution.version
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages. # for a list of supported languages.
...@@ -186,14 +209,49 @@ html_favicon = 'img/favicon.ico' ...@@ -186,14 +209,49 @@ html_favicon = 'img/favicon.ico'
# Output file base name for HTML help builder. # Output file base name for HTML help builder.
htmlhelp_basename = project_variable + u'_doc' htmlhelp_basename = project_variable + u'_doc'
# -- Post configuration -------------------------------------------------------- # -- Post configuration --------------------------------------------------------
# Included after all input documents
rst_epilog = """
.. |project| replace:: Bob
.. |version| replace:: %s
.. |current-year| date:: %%Y
""" % (version,)
# Default processing flags for sphinx # Default processing flags for sphinx
autoclass_content = 'class' autoclass_content = 'class'
autodoc_member_order = 'bysource' autodoc_member_order = 'bysource'
autodoc_default_flags = [ autodoc_default_flags = [
'members', 'members',
'undoc-members', 'undoc-members',
'inherited-members',
'show-inheritance', 'show-inheritance',
] ]
# For inter-documentation mapping:
intersphinx_mapping = {
'python': ('https://docs.python.org/%d.%d/' % sys.version_info[:2], None),
'six': ('https://six.readthedocs.io', None),
'setuptools': ('https://setuptools.readthedocs.io/en/latest/', None),
'bob.extension': ('http://www.idiap.ch/software/bob/docs/bob/bob.extension/stable/', None),
}
# We want to remove all private (i.e. _. or __.__) members
# that are not in the list of accepted functions
accepted_private_functions = ['__array__']
def member_function_test(app, what, name, obj, skip, options):
# test if we have a private function
if len(name) > 1 and name[0] == '_':
# test if this private function should be allowed
if name not in accepted_private_functions:
# omit privat functions that are not in the list of accepted private functions
return skip
else:
# test if the method is documented
if not hasattr(obj, '__doc__') or not obj.__doc__:
return skip
return False
def setup(app):
app.connect('autodoc-skip-member', member_function_test)
doc/development.rst 0 → 100644
+ 101
− 0
View file @ 26aca3c8
.. _bob.buildout.dev:
=========================
Developing bob.buildout
=========================
You can quickly test this package by running the following commands:
.. code-block:: sh
$ buildout
$ ./bin/nosetests -sv
Testing is limited to certain internal functionality. If you want to do an
extensive test and make sure changes are operating when you use this package to
create an environment for *another* package, please read on.
Cross-developing with another package
-------------------------------------
This is a chicken-and-egg problem as developing another package with a *new*
version of this package requires a working installation of ``bob.buildout``. In
order to break the loop, you'll need to first buildout with a simplified
version of ``buildout.cfg``, to tell buildout ``bob.buildout`` is being
developed alongside the test package. As an example, we'll develop
``bob.buildout`` in a ``bob.extension`` checkout:
.. code-block:: sh
$ git clone https://gitlab.idiap.ch/bob/bob.extension
$ cd bob.extension
# create the file first.cfg with the following contents:
$ cat first.cfg
[buildout]
parts =
develop = src/buildout
$ mkdir src
$ git clone https://gitlab.idiap.ch/bob/bob.buildout src/bob.buildout
Setup your base conda-environment as usual and the, run ``buildout -c
first.cfg``:
.. code-block:: sh
$ buildout -c first.cfg
The previous command should not download anything from PyPI_ and will create a
symbolic egg link in ``develop-eggs`` called ``bob.buildout.egg-link``. To make
sure your ``first.cfg`` bootstrap procedure worked, check there. Now, slightly
modify ``buildout.cfg`` from ``bob.extension`` to include a new line in the
``buildout.develop`` entry before ``.``, so the new buildout will also take the
bootstrapped buildout into consideration. It should look like this:
.. code-block:: sh
$ cat buildout.cfg
[buildout]
parts = scripts
develop = src/bob.buildout
.
eggs = bob.extension
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true
Now run buildout normally, against the modified ``buildout.cfg``:
.. code-block:: sh
$ buildout
...
This last step should provide you with a setup as performed by the
bleeding-edge version of bob.buildout you have checked-out on the ``src``
directory. You can modify it and re-run ``buildout`` until everything looks in
order.
Releasing bob.buildout
----------------------
This is a standard |project| package and therefore you **must** follow the
standard releasing procedure using ``bob_new_version.py`` or any more recent
script available for such purpose.
You may create a ``bob.extension``-based environment for such as per
instructions above or start from a pure conda-based install which contains
``bob.extension`` and run that script from it.
.. include:: links.rst
doc/guide.rst 0 → 100644
+ 218
− 0
View file @ 26aca3c8
.. bob.buildout.guide:
=========================================
Local development of |project| packages
=========================================
Very often, developers of |project| packages are confronted with the need to
clone repositories locally and develop installation/build and runtime code.
While it is possible to use conda_ for such, the use of `zc.buildout`_ offers
an quick and easy alternative to achieve this. It allows the creation of
isolated, directory-based python development environments that can be modulated
based on the development needs of the current package(s) one needs to work on.
The steps involved in creating a development environment are the following:
1. Checkout from gitlab_ the package the user wants to develop
2. Create a conda installation containing base packages that the current
package being developed requires
3. *Optionally*, create a buildout configuration that allows the
cross-development of packages
4. Run the application ``buildout`` to set-up the desired development
environment
This guide is a step-by-step guide in performing these.
Checking out |project| package sources
--------------------------------------
|project| packages are developed through Gitlab_. In order to checkout a
package, just use git_:
.. code-block:: sh
$ git clone https://gitlab.idiap.ch/bob/<package>
Where ``<package>`` is the package you want to develop. Various packages exist
in |project|'s gitlab_ instance.
Create a base conda-installation
--------------------------------
The base conda installation should contemplate all packages that you want to
develop against. This is typically listed in the package's
``requirements.txt``, but may also include test dependencies listed in
``test-requirements.txt`` depending on the package. The file
``conda/meta.yaml`` should be considered the canonical source for information
concerning package installation and deployment. After following `Bob's
installation`_ instructions, install all packages listed on the ``meta.yaml``
file using a single conda installation command.
For example, if the package one needs to develop lists build dependencies
``A``, ``B`` and ``C`` and test dependencies ``T`` and ``U``, the following
command-line should suffice once you followed `Bob's installation`_
instructions:
.. code-block:: sh
$ cd <package>
$ cat conda/meta.yaml #inspect contents and decide what to install
...
$ conda create -n dev A B C T U bob.buildout
...
$ source activate dev #ready to develop your package
**Optional** Use of not-yet-released conda packages
===================================================
When developing and testing new features, one often wishes to work against the
very latest, *bleeding edge*, available set of changes on dependent packages.
If that is the case, consider adding our `conda beta channel`_ alongside conda
channels available for download in your condarc_ file.
The location of ``.condarc`` is configurable, but it is often set in
``${HOME}/.condarc``. Read about condarc_ online if you are in doubt.
After our `conda beta channel`_ is included on your configuration, proceed as
above to create an environment with the latest dependencies instead of the
latest *stable* versions of each package.
**Optional** Automated environment creation
===========================================
It is possible to automate the above procedure using a script that tries to
automatically parse build and test requirements in ``conda/meta.yaml`` and
create the required development environment from scratch. This script lives in
the package ``bob.admin``. The first step is to checkout ``bob.admin``
alongside the package you're developing:
.. code-block:: sh
$ cd ..
$ ls
<PACKAGE>
$ git checkout https://gitlab.idiap.ch/bob/bob.admin
$ ls
<PACKAGE>
bob.admin
$ cd <PACKAGE>
.. note::
If you already have checked out ``bob.admin``, make sure to update it with
``git pull``. We're constantly making improvements to this package.
Once ``bob.admin`` is available alongside your package, make sure both
``conda`` and ``conda-build`` are installed **and updated** on the root
environment of your conda_ installation. The automated script requires conda_
version 4.4 or above and ``conda-build`` version 3 or above. It also requires
the package ``pyyaml`` to be installed on the root of your conda installation.
Follow this recipe to get all up-to-date and ready:
.. code-block:: sh
$ conda update -n root conda conda-build
...
$ conda install -n root pyyaml
...
.. note::
Notice the application ``conda`` is in my ``${PATH}`` and therefore the
shell can find it easily. **Make sure you do the same**.
Now that you're all set, just call the script
``bob.admin/conda/conda-bootstrapy`` and pass the name of the resulting
environment you'd like to create:
.. code-block:: sh
$ cd <PACKAGE>
$ ../bob.admin/conda/conda-bootstrap.py dev
...
This will parse the conda recipe from your package and create a new conda
environment on your conda installation called ``dev``. The environment ``dev``
contains all build *and* test dependencies required for your package. Activate
this environment and you're ready.
.. note::
By default, our script **will include** our `conda beta channel`_ while
creating your environment. You may modify the file
``bob.admin/conda/build-condarc`` if you'd like to include or remove
channels.
In this setup, we bypass your own condarc_ setup and use a stock version
provided with ``bob.admin``.
Running buildout
----------------
The last step is to create a hooked-up environment so you can quickly test
local changes to your package w/o necessarily creating a conda-package. This
step is the easiest:
.. code-block:: sh
$ cd <PACKAGE> #if that is not the case
$ source activate dev
$ buildout
...
zc.buildout_ works by modifying the load paths of scripts to find the correct
version of your package sources from the local checkout. After running,
buildout creates a directory called ``bin`` on your local package checkout. Use
the applications living there to develop your package. For example, if you need
to run the test suite:
.. code-block:: sh
$ ./bin/nosetests -sv
A python interpreter clone can be used to run interactive sessions:
.. code-block:: sh
$ ./bin/python
**Optional** Cross-development of dependencies
----------------------------------------------
From time to time, you may wish to cross-develop multiple projects at once. For
example, you may wish to develop ``bob.bio.face``, while *also* making
modifications to ``bob.bio.base``. In this case, you'll need to create a
buildout recipe (i.e., a new ``.cfg``) file that instructs buildout to also
checkout the sources for ``bob.bio.base`` while setting up the local structure
for ``bob.bio.face``. Follow our development guide from ``bob.extension`` at
:ref:`bob.extension` for more instructions on this step. Once your new ``.cfg``
is ready, use it like this setup:
.. code-block:: sh
$ buildout -c newrecipe.cfg
.. include:: links.rst
doc/index.rst
+ 51
− 8
View file @ 26aca3c8
.. vim: set fileencoding=utf-8 : .. vim: set fileencoding=utf-8 :
.. Andre Anjos <andre.anjos@idiap.ch>
.. Mon 17 Mar 09:23:45 2014 CET .. Mon 17 Mar 09:23:45 2014 CET
..
.. Copyright (C) 2011-2014 Idiap Research Institute, Martigny, Switzerland
.. _bob.buildout: .. _bob.buildout:
==================== ==============
Bob Buildout Rules Bob Buildout
==================== ==============
.. todolist:: This package includes an extension and a recipe to make it easy to create
local, directory-based complete development environments for |project|
packages. While |project| packages are distributed as conda-based installable
packages, the use of zc.buildout_ may greaty shorten development cycles as it
avoids the conda-build step, which can be somewhat lengthy (e.g. for packages
with C/C++ bindings).
zc.buildout_, unlike conda_, can *not* handle non-python-based packages. It is
therefore a tool to quickly create deployments or test changes on
**python-based distributable packages**. It follows that, in order to use
zc.buildout_, packages *need* to be python-based packages. Luckily, most of
|project| packages fit in this category, besides also being conda_ packages, so
zc.buildout_ can be used for development purposes in this context. Although
zc.buildout_ supports deployment, it is currently **not** used for this purpose
in the context of |project|. Our dependence list goes beyond Python-only
packages (e.g. ffmpeg or blitz++) and a full-stack software deployment cannot
be easily achieved relying only on this python development tool.
.. todo:: Write documentation of this package zc.buildout_ uses Python setuptools_ to create a directory structure that
contains scripts and dependencies which can defined in a *buildout recipe*.
Once a buildout recipe is interpreted, zc.buildout_ will make sure all listed
dependencies are satisfied and automatically instantiate scripts with changed
load paths that take in consideration all python packages. In order to satistfy
dependencies, zc.buildout_ may download missing python-based packages from
PyPI_, the Python Package Index. In the context of |project|-based package
development, we seldomly use this feature though.
Because zc.buildout_ is used for |project| package development since quite
sometime, most packages in the |project| ecosystem include a file called
``buildout.cfg`` that can be used by zc.buildout_ to build a quick,
*throw-away* development environment for the current package. A common
misconception is that one must checkout a |project| package in order to run
buildout. That is **not** true. zc.buildout_ is an independent development
tool that just deployes a python-based software stack based on a *recipe*. If
the *recipe* mentions the code of a package on the current directory, the setup
will include this. If it doesn't, then it won't. We typically don't use
zc.buildout_ to create deployments for |project| packages - we normally defer
this task to conda_ as it can handle non-pythonic dependencies in a much better
and uniform way. That said, it is possible to create python-only based
deployments using zc.buildout_ *without* being necessarily in the root
directory of a |project| package.
This document explains how to use zc.buildout_ and ``bob.buildout``'s recipe
and extension to construct development environments for your |project|
packages.
Documentation Documentation
...@@ -22,6 +60,11 @@ Documentation ...@@ -22,6 +60,11 @@ Documentation
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
guide
buildout
development
py_api
Indices and tables Indices and tables
------------------ ------------------
......
doc/links.rst
+ 9
− 7
View file @ 26aca3c8
.. vim: set fileencoding=utf-8 : .. vim: set fileencoding=utf-8 :
.. Andre Anjos <andre.anjos@idiap.ch> .. Tue 27 Feb 2018 09:11:11 CET
.. Tue 20 Mar 2012 08:57:32 CET
..
.. Copyright (C) 2011-2014 Idiap Research Institute, Martigny, Switzerland
.. _bob's website: https://www.idiap.ch/software/bob .. _bob's website: https://www.idiap.ch/software/bob
.. _bob: https://www.idiap.ch/software/bob .. _bob: https://www.idiap.ch/software/bob
.. _bob's installation: https://www.idiap.ch/software/bob/install
.. _bob packages: https://www.idiap.ch/software/bob/packages
.. _c++: http://www2.research.att.com/~bs/C++.html .. _c++: http://www2.research.att.com/~bs/C++.html
.. _conda: https://conda.io
.. _conda channel: https://www.idiap.ch/software/bob/conda/
.. _conda beta channel: https://www.idiap.ch/software/bob/conda/label/beta
.. _condarc: https://conda.io/docs/user-guide/configuration/use-condarc.html
.. _distutils: http://docs.python.org/distutils/ .. _distutils: http://docs.python.org/distutils/
.. _git: http://git-scm.com/ .. _git: http://git-scm.com/
.. _github: http://github.com/ .. _gitlab: https://gitlab.idiap.ch/bob/
.. _idiap: http://www.idiap.ch .. _idiap: http://www.idiap.ch
.. _ipython: http://ipython.scipy.org .. _ipython: http://ipython.scipy.org
.. _nose: https://nose.readthedocs.org/en/latest/ .. _nose: https://nose.readthedocs.org/en/latest/
.. _pep 386: http://www.python.org/dev/peps/pep-0386/ .. _pep 386: http://www.python.org/dev/peps/pep-0386/
.. _python: http://www.python.org .. _python: http://www.python.org
.. _pypi: http://pypi.python.org .. _pypi: http://pypi.python.org
.. _satellite packages: https://github.com/idiap/bob/wiki/Satellite-Packages .. _setuptools: https://setuptools.readthedocs.io
.. _setuptools: http://trac.edgewall.org/wiki/setuptools
.. _sphinx: http://sphinx.pocoo.org .. _sphinx: http://sphinx.pocoo.org
.. _zc.buildout: http://pypi.python.org/pypi/zc.buildout/ .. _zc.buildout: http://pypi.python.org/pypi/zc.buildout/
doc/nitpick-exceptions.txt 0 → 100644
+ 1
− 0
View file @ 26aca3c8
py:class zc.recipe.egg.egg.Scripts
doc/py_api.rst 0 → 100644
+ 33
− 0
View file @ 26aca3c8
.. _bob.buildout.api:
============
Python API
============
This section includes information for using the Python API of ``bob.buildout``.
Extension
---------
.. automodule:: bob.buildout.extension
Recipe
------
.. automodule:: bob.buildout.scripts
Other Modules
-------------
.. automodule:: bob.buildout.envwrapper
.. automodule:: bob.buildout.tools
.. automodule:: bob.buildout.script
.. automodule:: bob.buildout.python
.. automodule:: bob.buildout.dbpy
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment