Commit b27f3503 authored by André Anjos's avatar André Anjos 💬

Merge branch 'docedit' into 'master'

[docs] removed information about building python packages locally

See merge request !102
parents 7357d003 b113b80b
Pipeline #36569 passed with stages
in 4 minutes and 11 seconds
=========================
Additional considerations
=========================
Unit tests
----------
Writing unit tests is an important asset on code that needs to run in different platforms and a great way to make sure all is OK.
Test units are run with nose_.
To run the test units on your package call:
.. code-block:: sh
$ ./bin/nosetests -v
bob.example.library.test.test_reverse ... ok
----------------------------------------------------------------------
Ran 1 test in 0.253s
OK
This example shows the results of the tests in the ``bob.example.project`` package. Ideally, you should
write test units for each function of your package ...
.. note::
You should put additional packages needed for testing (e.g. ``nosetests``)
in the ``test-requirements.txt`` file.
Continuous integration (CI)
---------------------------
.. note::
This is valid for people at Idiap (or external bob contributors with access to Idiap's gitlab)
.. note::
Before going into CI, you should make sure that your pacakge has a gitlab repository.
If not, do the following in your package root folder:
.. code-block:: sh
$ git init
$ git remote add origin git@gitlab.idiap.ch:bob/`basename $(pwd)`
$ git add bob/ buildout.cfg COPYING doc/ MANIFEST.IN README.rst requirements.txt setup.py version.txt
$ git commit -m '[Initial commit]'
$ git push -u origin master
Copy the appropriate yml template for the CI builds:
.. code-block:: sh
# for pure python
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-python-only.yml > .gitlab-ci.yml
# for c/c++ extensions
$ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-cxx-extensions.yml | tr -d '\r' > .gitlab-ci.yml
Add the file to git:
.. code-block:: sh
$ git add .gitlab-ci.yml
The ci file should work out of the box. It is long-ish, but generic to any
package in the system.
You also need to enable the following options - through gitlab - on your project:
1. In the project "Settings" page, make sure builds are enabled
2. If you have a private project, check the package settings and make sure that
the "Deploy Keys" for our builders (all `conda-*` related servers) are
enabled
3. Visit the "Runners" section of your package settings and enable all conda
runners, for linux and macosx variants
4. Go into the "Variables" section of your package setup and **add common
variables** corresponding to the usernames and passwords for uploading
wheels and documentation tar balls to our (web DAV) server, as well as PyPI
packages. You can copy required values from [the "Variables" section of
bob.admin](https://gitlab.idiap.ch/bob/bob.admin/variables). N.B.: You
**must** be logged into gitlab to access that page.
5. Make sure to **disable** the service "Build e-mails" (those are very
annoying)
6. Setup the coverage regular expression under "CI/CD pipelines" to have the
value `^TOTAL.*\s+(\d+\%)$`, which is adequate for figuring out the output
of `coverage report`
Python package namespace
------------------------
We like to make use of namespaces to define combined sets of functionality that go well together.
Python package namespaces are `explained in details here <http://peak.telecommunity.com/DevCenter/setuptools#namespace-package>`_ together with implementation details.
For bob packages, we usually use the ``bob`` namespace, using several sub-namespaces such as ``bob.io``, ``bob.ip``, ``bob.learn``, ``bob.db`` or (like here) ``bob.example``.
The scripts you created should also somehow contain the namespace of the package. In our example,
the script is named ``bob_example_project_version.py``, reflecting the namespace ``bob.example``
Distributing your work
----------------------
To distribute a package, we recommend you use PyPI_.
`Python Packaging User Guide <https://packaging.python.org/>`_ contains details
and good examples on how to achieve this.
Moreover, you can provide a conda_ package for your PyPI_ package for easier
installation. In order to create a conda_ package, you need to create a conda_
recipe for that package.
For more detailed instructions on how to achieve this, please see the
guidelines on `bob.template <https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates>`_.
.. include:: links.rst
This diff is collapsed.
......@@ -38,12 +38,10 @@ Documentation
cplusplus_modules
cplusplus_library
documenting
additional
rc
framework
py_api
cpp_api
pip
Indices and tables
------------------
......
......@@ -41,3 +41,4 @@
.. _discuss: https://www.idiap.ch/software/bob/discuss
.. _numpydoc docstring guide: https://numpydoc.readthedocs.io/en/latest/format.html
.. _new package instructions: https://gitlab.idiap.ch/bob/bob.admin/tree/master/templates
.. _bob development tools: https://www.idiap.ch/software/bob/develop
Using pip for development
==========================
.. warning::
This guide is not complete and not recommended. You can just skip this
page. There are some problems when using pip and conda at the same time
which may lead to breaking **all your conda environments**. You can look at
the ``conda develop`` command for an alternative.
Since all |project| packages are distributed with `Setuptools`_, you can easily
develop them using pip_. Additionally, you can use conda_ to create separate
isolated environments for your different tasks.
.. note::
Keep this in mind: 1 task, 1 environment. Do not mix.
.. note::
This guide does not aim to duplicate the `pip's user guide`_. Please go
through that first to make sure you are confident in using pip_ before
continuing here.
Developing a |project| package with pip_ can be as easy as running the
following command while your current directory is the source of the package
that you want to develop:
.. code-block:: sh
pip install -e .
This will install the current package in your Python environment in an editable
mode. You can keep changing this package at its source directory and your
changes will be immediately reflected in your environment.
.. warning::
If you modify the contents of ``setup.py`` such as adding or removing
console script entry points, you need to run ``pip install -e .`` again.
Developing a C++ Package
------------------------
While developing a C++ package, the same ``pip install -e .`` command can be
used to compile the package again. Also, you may want to export some debugging
flags if you want to debug the current package:
.. code-block:: sh
export CFLAGS=$CFLAGS -O0 -g -DBOB_DEBUG -DBZ_DEBUG
export CXXFLAGS=$CXXFLAGS -O0 -g -DBOB_DEBUG -DBZ_DEBUG
pip install -e .
Developing Several Packages from Source
---------------------------------------
Often it is required not only to develop the current package but also to
develop several dependencies from source too. To do so, you can create a
requirements file (usually named ``requirements.txt``) and run the following
command:
.. code-block:: sh
pip install -r requirements.txt
while the content of your ``requirements.txt`` file lists the packages that the
current package both depends on and the dependencies that you want to develop.
For example, the following ``requirements.txt`` file can be used to develop
:py:ref:`bob.io.image <bob.io.image>`, together with **all** of its direct and
indirect dependencies::
setuptools
-egit+https://gitlab.idiap.ch/bob/bob.extension.git#egg=bob.extension
-egit+https://gitlab.idiap.ch/bob/bob.blitz.git#egg=bob.blitz
-egit+https://gitlab.idiap.ch/bob/bob.core.git#egg=bob.core
-egit+https://gitlab.idiap.ch/bob/bob.io.base.git#egg=bob.io.base
matplotlib
However because of a limitation of pip_, you need to install the |project|
dependencies that contain C++ code both in order and one by one. This means
that ``pip install -r requirements.txt`` will not work in this case. Instead,
you can run the following command:
.. code-block:: sh
tr '\n' '\0' < requirements.txt | xargs -0 -n 1 pip install
.. warning::
For the ``tr '\n' '\0' < requirements.txt | xargs -0 -n 1 pip install``
command to work, you should not have any spaces in your
``requirements.txt`` file (notice that ``-egit+https://...`` has no spaces)
and your |project| dependencies should be listed in their order of
dependencies.
.. include:: links.rst
......@@ -132,10 +132,11 @@ Building your package
---------------------
To be able to use the package, we first need to build and install it locally.
This is explained in detail in :ref:`bob.extension.build_locally`.
This is explained in detail in `bob development tools`_.
The buildout configuration file of the package looks like:
.. code-block:: cfg
.. code-block:: ini
[buildout]
parts = scripts
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment