Commit 26aca3c8 authored by André Anjos's avatar André Anjos 💬
Browse files

Merge branch 'issue-25' into 'master'

Issue 25

Closes #25

See merge request !29
parents 71e6ff30 4b5354f3
Pipeline #18290 passed with stages
in 6 minutes and 36 seconds
.. vim: set fileencoding=utf-8 :
.. Thu 30 Jan 08:46:53 2014 CET
.. Tue 27 Feb 2018 09:06:28 CET
.. image:: http://img.shields.io/badge/docs-stable-yellow.png
.. image:: https://img.shields.io/badge/docs-stable-yellow.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/stable/index.html
.. image:: http://img.shields.io/badge/docs-latest-orange.png
.. image:: https://img.shields.io/badge/docs-latest-orange.svg
:target: https://www.idiap.ch/software/bob/docs/bob/bob.buildout/master/index.html
.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/build.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
.. image:: https://gitlab.idiap.ch/bob/bob.buildout/badges/master/coverage.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout/commits/master
.. image:: https://img.shields.io/badge/gitlab-project-0000c0.svg
:target: https://gitlab.idiap.ch/bob/bob.buildout
.. image:: http://img.shields.io/pypi/v/bob.buildout.png
.. image:: https://img.shields.io/pypi/v/bob.buildout.svg
:target: https://pypi.python.org/pypi/bob.buildout
......@@ -18,157 +20,17 @@
===================================
This package is part of the signal-processing and machine learning toolbox
Bob_. It contains a number of ``zc.buildout`` recipes for simplifying builds of
Bob_ packages.
C++/Python Extension
--------------------
This extension allows you to compile C/C++ extensions that depend on each other
using a buildout. It assures that eggs living in both ``develop-eggs`` and
``eggs`` are on your path before building the packages in the ``develop``
section. By using this extension you can drop the use of the local recipe
``bob.buildout:develop``, which should be considered deprecated.
Supported Options
=================
verbose
If set, buildout it will output the compilation commands while compiling the
module.
prefixes
A list of directories where this recipe will look for installed software,
such as compiled libraries and header files. It is the same as setting the
environment variable ``BOB_PREFIX_PATH`` to a list of paths containing
externally installed software. As a side-effect, setting ``BOB_PREFIX_PATH``
also sets, internally, ``PKG_CONFIG_PATH`` to a list of directories following
where to search for pkg-config files.
debug
If set, the module will be compiled with debugging symbols and with
optimization turned off. If ``debug`` is set to ``true``, this is equivalent
to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
-g``. If it is set to ``false``, it is the same as appending ``-O3 -g0``
instead.
environ
The name of a section on your configuration file that contains the names and
values of environment variables that should be used through the build. This
section is named, by default, ``environ``.
If a section named ``environ`` exists, it is read and the environment
variables are set **before** the specified eggs are developed. You can use
variable substitution on this section. Here is an an example::
[environ]
CFLAGS = '-O0 -g -DNDEBUG'
CXXFLAGS = ${CFLAGS}
Notice there is some functionality overlap between the previous flags and the
use of section ``environ``. While it is more flexible, you must understand
the consequences of setting both ``prefixes`` and ``debug``, together with
``environ``. The rule is simple: values set on the ``environ`` section have
**precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
to those listed in ``prefixes``, if that is also set.
Multi-Script Installer
----------------------
This recipe installs **all** most used scripts and interpreter proxies for your
package. It will look at the ``buildout`` section entry called ``prefixes``,
that potentially lists prefixes that should be **prepended** to the default
python environment. In these prefixes, it will look for standard python
directories. If one or more are found, these paths are **prepended** into
the resulting scripts generated by this recipe and eggs will be searched on
those locations prioritarily.
By default, this recipe will use the eggs defined at the ``buildout`` section
called ``eggs``, but that can be overriden locally. It generates these scripts:
python
A pre-configured python interpreter
gdb-python
A pre-configured python interpreter, prefixed with ``gdb`` to make debugging
easier. Use it like you use ``python``.
nosetests
A test runner called ``nosetests`` will be created on the bin directory of
buildout.
coverage
A test coverage application called ``coverage`` will be created on the bin
directory of buildout.
sphinx
Several sphinx utilities will be created on the bin directory of buildout.
package scripts
Package scripts will be created taking into account the ``prefixes``
established for this section or globally (as a second priority).
To use this recipe, you just have to simply do::
[scripts]
recipe = bob.buildout:scripts
Common Supported Options
========================
The recipe supports the following options:
prefixes
A list of directories where this recipe will look for subdirectories with
the stem ``lib/python*/site-packages``. All directories matching this
condition are appended to the search paths. If not given, the value of this
property defaults to ``buildout.prefixes``. Both can be empty, which makes
this recipe default to using standard available paths.
eggs
The eggs option specifies a list of eggs to use for **building** this
package. Each string must be given on a separate line. If not given, the
value of this property defaults to ``buildout.eggs``.
dependent-scripts
If set to the string ``true``, scripts will be generated for all required
eggs in addition to the eggs specifically named.
interpreter
The name of a script to generate that allows access to a Python interpreter
that has the path set based on the eggs installed. If you don't specify
anything, the default value ``python`` will be used.
extra-paths
Extra paths to be appended in a generated script. To prepend, using the
``prefixes`` entry.
nose-flags
These are extra flags that are **appended** to the given ``nosetests``
command line, automatically. Use this to preset arguments you like running
all the time like ``-v``, for example.
Bob_. It contains a number of ``zc.buildout`` recipes for simplifying
development of Bob_ packages.
Installation
------------
Follow our `installation`_ instructions. Then, using the Python interpreter
provided by the distribution, bootstrap and buildout this package::
Complete Bob's `installation`_ instructions. Then, to install this package,
run::
$ python bootstrap-buildout.py
$ ./bin/buildout
$ conda install bob.buildout
Contact
......@@ -180,5 +42,5 @@ development `mailing list`_.
.. Place your references here:
.. _bob: https://www.idiap.ch/software/bob
.. _installation: https://gitlab.idiap.ch/bob/bob/wikis/Installation
.. _mailing list: https://groups.google.com/forum/?fromgroups#!forum/bob-devel
.. _installation: https://www.idiap.ch/software/bob/install
.. _mailing list: https://www.idiap.ch/software/bob/discuss
.. bob.buildout.recipes:
=======================
Extension and Recipes
=======================
This package is composed of an extension to zc.buildout_ and a recipe. We
recommend you use both during the creation of your own package-based recipes.
Here is a typical ``buildout.cfg`` file that can be used to setup the
development of your own package, and which we normally ship with all
|project|-based packages:
.. code-block:: ini
[buildout]
parts = scripts
develop = .
eggs = <PACKAGE>
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
Replace ``<PACKAGE>`` by your package name and you should be ready to run the
``buildout`` application. If you're curious about zc.buildout_ and how to
structure your recipe to take full advantage of it, we advise you seek
documentation on that package on its website or by searching the internet.
The above setup will include all required material for building both simple
python or python/C/C++ packages from |project|.
By using extension ``bob.buildout`` on the ``buildout`` section of your recipe,
you ensure the current package will be built taking into consideration all
required environment/compiler settings required by |project|. This extension is
strictly required for packages containing C/C++ bindings, but it is harmless to
include it in Python-only packages.
The section ``scripts`` define a list of scripts that will be installed on the
``bin`` directory. By default, we make sure all packages listed in
``buildout.eggs`` are available, as well as typical packages required for
development such as ``sphinx``, for documentation building, ``nose`` for
running your test suite and ``coverage``, for code coverage reporting. You may
augment the ``eggs`` entry on the ``buildout`` section if you'd like further
packages to be installed on your development environment. It is your call. An
example package that is often listed there is ``ipdb`` - the iPython based
python debugger. The syntax of the ``eggs`` entry on the ``buildout`` section
is one package per line.
The entry ``develop`` on the ``buildout`` section indicates to buildout the
root directories of packages that it should take, prioritarily, before
considering potential versions installed on base python (conda) environment. It
typically says just ``.`` (dot), as we're typically willing to *only* make
buildout aware of our local checkout. It can include more directories (one per
line), if we'd like to cross-develop more packages. That is what we do, for
example, when using the extension ``mr.developer`` (see:
:ref:`bob.extension` development). For each package listed in ``develop``,
buildout will run the equivalent of ``cd <dir> && python setup.py develop`` on
each package **before** trying to resolve the dependencies in ``eggs``. It will
then consider those locally installed packages in final load-path scheme for
your applications.
In case you're curious, the ``buildout.newest`` flag is internal to
zc.buildout_. It instructs the setuptools_ machinery to consider or not
versions of packages currently installed. If ``newest=true``, then only the
absolute newest packages in PyPI_ can satisfy the build. By default, we let
this setting to ``false`` indicating that what is currently installed will be
sufficient. Only change it if you know the implications. Note, for example, in
this case, you may not be testing anymore, exclusively, against a
|project|-agreed set of dependencies.
The remainder flags and options are explained next.
Supported extension options
---------------------------
Here is a list of supported options for the ``bob.buildout``. They should
appear in the ``buildout`` section of your recipe. They are considered for as
long as ``bob.buildout`` is listed as an extension.
``verbose``
If set, buildout it will output the compilation commands while compiling the
module. This flag should go into the ``buildout`` section of your recipe.
Accepted values are ``true`` or ``false``.(the default).
``prefixes``
A list of directories where this recipe will look for installed software,
such as compiled libraries and header files. It is important if you're
compiling a package that requires external software such as ffmpeg or blitz++
header files to be available. It is the same as setting the environment
variable ``BOB_PREFIX_PATH`` to a list of paths containing externally
installed software. As a side-effect, setting ``BOB_PREFIX_PATH`` also sets,
internally, ``PKG_CONFIG_PATH`` to a list of directories following where to
search for pkg-config files.
You typically don't need to set this manually since ``bob.extension`` is
smart enough to figure out where to find external libraries. This is even
more true if you're using conda-based installations as the bootstrapping
environment of your buildout.
``debug``
If set, the module will be compiled with debugging symbols and with
optimization turned off. If ``debug`` is set to ``true``, this is equivalent
to appending the environment variables ``CFLAGS`` and ``CXXFLAGS`` with ``-O0
-g``. By default, the value is ``false``. This setting is advised if you are
compiling python C/C++ bindings and would like to debug C/C++ code locally.
``environ``
The name of another section on your configuration file that contains the
names and values of environment variables that should be set before any
possible build takes place. This section is named, by default, ``environ``.
That is, if you provide a section named ``environ`` on your buildout recipe,
then you don't need to explicitly specify its name using this flag.
If a section named ``environ`` (or whatever lists the ``environ`` override on
the ``buildout`` section of your recipe) exists, it is read and the
environment variables are set **before** the specified packages are built.
You can use variable substitution on this section. Here is an an example::
[environ]
CFLAGS = '-O0 -g -DNDEBUG'
CXXFLAGS = ${CFLAGS}
Notice there is some functionality overlap between the previous flags and the
use of section ``environ``. While it is more flexible, you must understand
the consequences of setting both ``prefixes`` and ``debug``, together with
``environ``. The rule is simple: values set on the ``environ`` section have
**precedence** to ``debug`` and ``prefixes``. If you set ``debug`` and
``CFLAGS`` (or ``CXXFLAGS``) in the ``environ`` section, for example, then
the values on the final ``CFLAGS`` variable would be ``-O0 -g`` followed by
``environ``'s ``CFLAGS`` settings. Analogously, the paths defined by
``environ``'s ``BOB_PREFIX_PATH`` and ``PKG_CONFIG_PATH`` are **prepended**
to those listed in ``prefixes``, if that is also set.
The ``scripts`` recipe
----------------------
By using the recipe ``bob.buildout:scripts`` on one of the sections of your
recipe, you ensure the scripts generated by the recipe will be built taking
into consideration all installed packages from your base python environment
(typically a conda-based installation). If you don't use the
``bob.buildout:scripts`` recipe, zc.buildout_, by default, assumes no packages
are availabe on the python installation and may download/recompile all
dependencies from scratch.
By default, this recipe will use the eggs defined at the ``buildout`` section
called ``eggs``, but that can be overriden locally. It generates these scripts:
``python``
A pre-configured python interpreter taking into consideration all eggs and
development packages on your recipe.
``gdb-python`` or ``lldb-python``
A pre-configured python interpreter, prefixed with ``gdb`` (or ``lldb`` on a
MacOS system) to make debugging easier. Use it like you use ``python``.
``nosetests``
A test runner called ``nosetests`` will be created on the bin directory of
buildout.
``coverage``
A test coverage application called ``coverage`` will be created on the
``bin`` directory of buildout.
``sphinx-build`` and friends
Several sphinx utilities will be created on the bin directory of buildout.
Other package scripts
Package scripts will be created taking into account the ``prefixes``
established for this section or globally (as a second priority).
Supported recipe options
========================
The ``scripts`` recipe supports the following options (mostly for experts -
read don't use them unless you know what you're doing):
``prefixes``
Overrides or sets the list of prefixes in ``buildout.prefixes``. If not
given, the value of this property defaults to ``buildout.prefixes``. Both can
be empty, which makes this recipe default to using standard available paths.
``eggs``
The eggs option specifies a list of eggs to use for **building** the scripts
of this recipe. Each string must be given on a separate line. If not given,
the value of this property defaults to ``buildout.eggs``.
``dependent-scripts``
If set to the string ``true``, scripts will be generated for all required
eggs in addition to the eggs specifically named. By default, it is ``false``
to avoid the clutter it may cause on very high-level packages, with numerous
dependencies exporting scripts.
``interpreter``
The name of a script to generate that allows access to a Python interpreter
that has the path set based on the eggs installed. If you don't specify
anything, the default value ``python`` will be used.
``extra-paths``
Extra paths to be appended in a generated script. To prepend, use the
``prefixes`` entry.
``nose-flags``
These are extra flags that are **appended** to the given ``nosetests``
command line, automatically. Use this to preset arguments you like running
all the time like ``-v``, for example.
.. include:: links.rst
......@@ -4,6 +4,7 @@
import os
import sys
import glob
import pkg_resources
# -- General configuration -----------------------------------------------------
......@@ -27,6 +28,25 @@ extensions = [
'sphinx.ext.mathjax',
]
# Be picky about warnings
nitpicky = True
# Ignores stuff we can't easily resolve on other project's sphinx manuals
nitpick_ignore = []
# Allows the user to override warnings from a separate file
if os.path.exists('nitpick-exceptions.txt'):
for line in open('nitpick-exceptions.txt'):
if line.strip() == "" or line.startswith("#"):
continue
dtype, target = line.split(None, 1)
target = target.strip()
try: # python 2.x
target = unicode(target)
except NameError:
pass
nitpick_ignore.append((dtype, target))
# Always includes todos
todo_include_todos = True
......@@ -57,14 +77,17 @@ project = u'bob.buildout'
import time
copyright = u'%s, Idiap Research Institute' % time.strftime('%Y')
# Grab the setup entry
distribution = pkg_resources.require(project)[0]
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = open('../version.txt').read().strip()
version = distribution.version
# The full version, including alpha/beta/rc tags.
release = version
release = distribution.version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......@@ -186,14 +209,49 @@ html_favicon = 'img/favicon.ico'
# Output file base name for HTML help builder.
htmlhelp_basename = project_variable + u'_doc'
# -- Post configuration --------------------------------------------------------
# Included after all input documents
rst_epilog = """
.. |project| replace:: Bob
.. |version| replace:: %s
.. |current-year| date:: %%Y
""" % (version,)
# Default processing flags for sphinx
autoclass_content = 'class'
autodoc_member_order = 'bysource'
autodoc_default_flags = [
'members',
'undoc-members',
'inherited-members',
'show-inheritance',
]
# For inter-documentation mapping:
intersphinx_mapping = {
'python': ('https://docs.python.org/%d.%d/' % sys.version_info[:2], None),
'six': ('https://six.readthedocs.io', None),
'setuptools': ('https://setuptools.readthedocs.io/en/latest/', None),
'bob.extension': ('http://www.idiap.ch/software/bob/docs/bob/bob.extension/stable/', None),
}
# We want to remove all private (i.e. _. or __.__) members
# that are not in the list of accepted functions
accepted_private_functions = ['__array__']
def member_function_test(app, what, name, obj, skip, options):
# test if we have a private function
if len(name) > 1 and name[0] == '_':
# test if this private function should be allowed
if name not in accepted_private_functions:
# omit privat functions that are not in the list of accepted private functions
return skip
else:
# test if the method is documented
if not hasattr(obj, '__doc__') or not obj.__doc__:
return skip
return False
def setup(app):
app.connect('autodoc-skip-member', member_function_test)
.. _bob.buildout.dev:
=========================
Developing bob.buildout
=========================
You can quickly test this package by running the following commands:
.. code-block:: sh
$ buildout
$ ./bin/nosetests -sv
Testing is limited to certain internal functionality. If you want to do an
extensive test and make sure changes are operating when you use this package to
create an environment for *another* package, please read on.
Cross-developing with another package
-------------------------------------
This is a chicken-and-egg problem as developing another package with a *new*
version of this package requires a working installation of ``bob.buildout``. In
order to break the loop, you'll need to first buildout with a simplified
version of ``buildout.cfg``, to tell buildout ``bob.buildout`` is being
developed alongside the test package. As an example, we'll develop
``bob.buildout`` in a ``bob.extension`` checkout:
.. code-block:: sh
$ git clone https://gitlab.idiap.ch/bob/bob.extension
$ cd bob.extension
# create the file first.cfg with the following contents:
$ cat first.cfg
[buildout]
parts =
develop = src/buildout
$ mkdir src
$ git clone https://gitlab.idiap.ch/bob/bob.buildout src/bob.buildout
Setup your base conda-environment as usual and the, run ``buildout -c
first.cfg``:
.. code-block:: sh
$ buildout -c first.cfg
The previous command should not download anything from PyPI_ and will create a
symbolic egg link in ``develop-eggs`` called ``bob.buildout.egg-link``. To make
sure your ``first.cfg`` bootstrap procedure worked, check there. Now, slightly
modify ``buildout.cfg`` from ``bob.extension`` to include a new line in the
``buildout.develop`` entry before ``.``, so the new buildout will also take the
bootstrapped buildout into consideration. It should look like this:
.. code-block:: sh
$ cat buildout.cfg
[buildout]
parts = scripts
develop = src/bob.buildout
.
eggs = bob.extension
extensions = bob.buildout
newest = false
verbose = true
[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true
Now run buildout normally, against the modified ``buildout.cfg``:
.. code-block:: sh
$ buildout
...
This last step should provide you with a setup as performed by the
bleeding-edge version of bob.buildout you have checked-out on the ``src``
directory. You can modify it and re-run ``buildout`` until everything looks in
order.
Releasing bob.buildout
----------------------
This is a standard |project| package and therefore you **must** follow the
standard releasing procedure using ``bob_new_version.py`` or any more recent
script available for such purpose.
You may create a ``bob.extension``-based environment for such as per
instructions above or start from a pure conda-based install which contains
``bob.extension`` and run that script from it.
.. include:: links.rst
.. bob.buildout.guide:
=========================================
Local development of |project| packages
=========================================
Very often, developers of |project| packages are confronted with the need to
clone repositories locally and develop installation/build and runtime code.
While it is possible to use conda_ for such, the use of `zc.buildout`_ offers
an quick and easy alternative to achieve this. It allows the creation of
isolated, directory-based python development environments that can be modulated
based on the development needs of the current package(s) one needs to work on.
The steps involved in creating a development environment are the following: