From 80add7a63e56f8eafd7876591ca494c09cf3cde5 Mon Sep 17 00:00:00 2001
From: Zohreh MOSTAANI <zohreh.mostaani@idiap.ch>
Date: Mon, 16 Sep 2019 19:20:10 +0200
Subject: [PATCH] [docs] more modification to include all the information about
building new packages but not very beautiful yet
---
doc/development.rst | 207 +++++++++++++++++++++++++++++------------
doc/guide.rst | 219 --------------------------------------------
doc/index.rst | 1 -
3 files changed, 147 insertions(+), 280 deletions(-)
delete mode 100644 doc/guide.rst
diff --git a/doc/development.rst b/doc/development.rst
index 3f5a41b2..74a836a3 100644
--- a/doc/development.rst
+++ b/doc/development.rst
@@ -10,7 +10,7 @@ It is recommended to create isolated environments to develop new projects using
their git_ source and build proper isolated environments to develop them.
TLDR
-----
+====
Suppose you want to develop two packages, ``bob.extension`` and ``bob.blitz``,
locally:
@@ -24,44 +24,44 @@ locally:
$ conda config --add channels defaults
$ conda config --add channels https://www.idiap.ch/software/bob/conda
-* Create an isolated environment for the task.
+* Install `bob.devtools` on a conda environment (create a new environment or install it on your base)
.. code-block:: sh
- $ conda create --copy -n awesome-project \
- python=3 bob-devel bob-extras
- # bob-devel has all of our dependencies but no bob packages themselves and
- # bob-extras has all of our bob packages.
- $ source activate awesome-project
+ $ conda create -n bdt -c https://www.idiap.ch/software/bob/conda bob.devtools
-* Create a folder with the following buildout configuration file.
+* Create an empty bob package using `bdt new` command:
.. code-block:: sh
- $ mkdir awesome-project
- $ cd awesome-project
- $ vi buildout.cfg
+ $ conda activate bdt
+ $ bdt new -vv bob/bob.awesome-project author_name author_email
+ $ cd bob.awesome-project
+
+* Edit `buildout.cfg` file to include the packages you want to develop.
.. code-block:: guess
[buildout]
parts = scripts
+ develop = src/bob.extension
+ src/bob.blitz
+ .
+
+ eggs = bob.extension
+ bob.blitz
+ bob.awesome-project
+
extensions = bob.buildout
mr.developer
+ auto-checkout = *
+
newest = false
verbose = true
debug = false
- auto-checkout = *
-
- develop = src/bob.extension
- src/bob.blitz
-
- eggs = bob.extension
- bob.blitz
-
[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true
@@ -73,6 +73,21 @@ locally:
; bob.extension = git git@gitlab.idiap.ch:bob/bob.extension.git
; bob.blitz = git git@gitlab.idiap.ch:bob/bob.blitz.git
+* Optionally add the packages you want to have in your local installation but not developing them to `eggs` in the `buildout.cfg` file and the `requirements/host` section of the `conda/meta.yaml` file.
+
+* Create an isolated environment for the your project.
+
+.. code-block:: sh
+
+ $ bdt create --stable -vv awesome-project
+ $ source activate awesome-project
+
+* Install necessary package for compiling bob packages
+
+.. code-block:: sh
+
+ $ conda install gcc_linux-64 gxx_linux-64
+
* Run buildout and check if your desired package is being imported from the
``awesome-project/src`` folder.
@@ -118,11 +133,16 @@ Optionally:
$ ../../bin/sphinx-build -aEn doc sphinx # make sure it finishes without warnings.
$ firefox sphinx/index.html # view the docs.
+.. note::
+
+ Sometimes when you are calling a function not interactively it is not acting normally. In that case import pkgsources before importing your package. It is a known issue and we are working on it.
+
+
.. bob.devtools.local_development:
-Local development of a Bob package
-==================================
+Local development of an existing Bob package
+============================================
Checking out |project| package sources
@@ -155,7 +175,7 @@ Before proceeding, you need to make sure that you already have a conda_ environm
$ conda activate dev
-Now you have an isolated conda environment with proper channels set. For more information about conda channels refer to `conda channel documentation`_.
+Now you have an isolated conda environment with proper channels set. Add info about what bdt create does! For more information about conda channels refer to `conda channel documentation`_.
.. note::
@@ -226,9 +246,11 @@ And you can install new necessary packages using conda:
If you want to debug a package regarding the issues showing on the ci you can use `bob.devtools`. Make sure the conda environment containing `bob.devtools` is activated.
- $ cd <package>
- $ conda activate bdt
- $ bdt local build
+ .. code-block:: sh
+
+ $ cd <package>
+ $ conda activate bdt
+ $ bdt local build
One important advantage of using conda_ and zc.buildout_ is that it does
@@ -237,65 +259,132 @@ Furthermore, you will be able to create distributable environments for each
project you have. This is a great way to release code for laboratory exercises
or for a particular publication that depends on |project|.
-.. _bob.devtools.build_locally:
+.. _bob.devtools.build_multi_package:
-Building packages locally
--------------------------
+Local development of a new Bob package
+======================================
-To be able to develop a package, we first need to build and install it locally.
-While developing a package, you need to install your package in *development*
-mode so that you do not have to re-install your package after every change that
-you do in the source. zc.buildout_ allows you to exactly do that.
-
-.. note::
- zc.buildout_ will create another local environment from your conda_
- environment but unlike conda_ environments this environment is not isolated
- rather it inherits from your conda_ environment. This means you can still
- use the libraries that are installed in your conda_ environment.
- zc.buildout_ also allows you to install PyPI_ packages into your
- environment. You can use it to install some Python library if it is not
- available using conda_. Keep in mind that to install a library you should
- always prefer conda_ but to install your package from source in
- *development* mode, you should use zc.buildout_.
-
-zc.buildout_ provides a ``buildout`` command. ``buildout`` takes as input a
-"recipe" that explains how to build a local working environment. The recipe, by
-default, is stored in a file called ``buildout.cfg``. Let\'s create an example
-one in your project folder that will allow you to develop ``bob.extension``
-from source:
+It so happens that you want to develop several bob.packages against each other for your project. In this case you can make a new package using `bdt` commands and develop it. You need to activate your conda environment with bob.devtools installed in it.
.. code-block:: sh
+
+ $ conda activate bdt
+ $ bdt new -vv bob/bob.awesome-project author_name author_email
- $ mkdir awesome-project
- $ cd awesome-project
- # checkout bob.extension from source and put it in a folder called `src`
- $ git clone https://gitlab.idiap.ch/bob/bob.extension src/bob.extension
+This command will create a new bob package named "awesome-project" that includes the correct anatomy but it doesn't have information about the packages you are depending on.
-Create a file named ``buildout.cfg`` in the ``awesome-project`` folder with the
-following contents:
+In the root of your project there is a file `buildout.cfg` which should look like:
.. code-block:: guess
[buildout]
parts = scripts
+ develop = .
+ eggs = bob.awesome-project
extensions = bob.buildout
newest = false
verbose = true
- debug = false
+
+ [scripts]
+ recipe = bob.buildout:scripts
+ dependent-scripts = true
+
+
+Let's assume you need to develop two packages at the same time, `bob.extension` and `bob.blitz`.
+
+You need to add these packages to the `buildout.cfg` file in the newly created folder.
+
+.. code-block:: guess
+
+ [buildout]
+ parts = scripts
develop = src/bob.extension
+ src/bob.blitz
+ .
+
eggs = bob.extension
+ bob.blitz
+ bob.awesome-project
+
+ extensions = bob.buildout
+ mr.developer
+
+ auto-checkout = *
+ newest = false
+ verbose = true
[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true
-Then, invoke buildout:
+ [sources]
+ bob.extension = git git@gitlab.idiap.ch:bob/bob.extension.git
+ bob.blitz = git git@gitlab.idiap.ch:bob/bob.blitz.git
+
+
+When you build your new package the dependent packages (in this example `bob.extension` and `bob.blitz`) will be checked out on folder `src` in the root of your project.
+
+Before you proceed with building your package you need to make a new isolated conda environment.
+
+.. code-block:: sh
+
+ $ cd bob.awesome-project
+ $ conda activate bdt
+ $ bdt create --stable -vv awesome-project
+ $ conda activate awesome-project
+
+Now you have a conda environment that is using *stable* channels to install bob packages. Some of bob packages need compilers to be build, so before running buildout you need to install the following packages:
.. code-block:: sh
+
+ $ conda install gcc_linux-64 gxx_linux-64
+
+.. note::
+ for macos you need to install the following:
+
+ .. code-block:: sh
+
+ $ conda install ??
+
+Now you can run buildout as usual.
+.. code-block:: sh
+
$ buildout
+.. note::
+
+ sometimes you may need some of bob.packages available in your local directory without necessarily developing them. In that case first install it on your conda environment using `conda install` command. Then add it to the `eggs` section in your `buildout.cfg` file and then run `buildout`.
+
+ If you knew beforehand what are those packages that you need you can add them to the `conda/meta.yaml` file and then create the conda environment using `bdt create` command which installs those packages automatically.
+
+
+
+
+
+Some notes on buildout
+----------------------
+
+To be able to develop a package, we first need to build and install it locally.
+While developing a package, you need to install your package in *development*
+mode so that you do not have to re-install your package after every change that
+you do in the source. zc.buildout_ allows you to exactly do that.
+
+.. note::
+ zc.buildout_ will create another local environment from your conda_
+ environment but unlike conda_ environments this environment is not isolated
+ rather it inherits from your conda_ environment. This means you can still
+ use the libraries that are installed in your conda_ environment.
+ zc.buildout_ also allows you to install PyPI_ packages into your
+ environment. You can use it to install some Python library if it is not
+ available using conda_. Keep in mind that to install a library you should
+ always prefer conda_ but to install your package from source in
+ *development* mode, you should use zc.buildout_.
+
+zc.buildout_ provides a ``buildout`` command. ``buildout`` takes as input a
+"recipe" that explains how to build a local working environment. The recipe, by
+default, is stored in a file called ``buildout.cfg``.
.. note::
Buildout by default looks for ``buildout.cfg`` in your current folder and
@@ -306,8 +395,6 @@ Then, invoke buildout:
$ buildout -c develop.cfg
-The buildout command with the configuration file above will install
-``bob.extension`` in *development mode* in your local buildout environment.
.. important::
Once ``buildout`` runs, it creates several executable scripts in a local
@@ -490,6 +577,6 @@ can access all packages that you have developed, including your own package:
Everything is now setup for you to continue the development of the packages.
Moreover, you can learn more about |project| packages and learn to create new
-ones in :doc:`pure_python`.
+ones in .
.. include:: links.rst
diff --git a/doc/guide.rst b/doc/guide.rst
deleted file mode 100644
index f67ebd86..00000000
--- a/doc/guide.rst
+++ /dev/null
@@ -1,219 +0,0 @@
-.. bob.buildout.guide:
-
-=========================================
- Local development of |project| packages
-=========================================
-
-Very often, developers of |project| packages are confronted with the need to
-clone repositories locally and develop installation/build and runtime code.
-While it is possible to use conda_ for such, the use of `zc.buildout`_ offers
-an quick and easy alternative to achieve this. It allows the creation of
-isolated, directory-based python development environments that can be modulated
-based on the development needs of the current package(s) one needs to work on.
-
-The steps involved in creating a development environment are the following:
-
-1. Checkout from gitlab_ the package the user wants to develop
-2. Create a conda installation containing base packages that the current
- package being developed requires
-3. *Optionally*, create a buildout configuration that allows the
- cross-development of packages
-4. Run the application ``buildout`` to set-up the desired development
- environment
-
-This guide is a step-by-step guide in performing these.
-
-
-Checking out |project| package sources
---------------------------------------
-
-|project| packages are developed through Gitlab_. In order to checkout a
-package, just use git_:
-
-
-.. code-block:: sh
-
- $ git clone https://gitlab.idiap.ch/bob/<package>
-
-
-Where ``<package>`` is the package you want to develop. Various packages exist
-in |project|'s gitlab_ instance.
-
-
-Create a base conda-installation
---------------------------------
-
-The base conda installation should contemplate all packages that you want to
-develop against. This is typically listed in the package's
-``requirements.txt``, but may also include test dependencies listed in
-``test-requirements.txt`` depending on the package. The file
-``conda/meta.yaml`` should be considered the canonical source for information
-concerning package installation and deployment. After following `Bob's
-installation`_ instructions, install all packages listed on the ``meta.yaml``
-file using a single conda installation command.
-
-For example, if the package one needs to develop lists build dependencies
-``A``, ``B`` and ``C`` and test dependencies ``T`` and ``U``, the following
-command-line should suffice once you followed `Bob's installation`_
-instructions:
-
-
-.. code-block:: sh
-
- $ cd <package>
- $ cat conda/meta.yaml #inspect contents and decide what to install
- ...
- $ conda create -n dev A B C T U bob.buildout
- ...
- $ conda activate dev #ready to develop your package
-
-
-**Optional** Use of not-yet-released conda packages
-===================================================
-
-When developing and testing new features, one often wishes to work against the
-very latest, *bleeding edge*, available set of changes on dependent packages.
-If that is the case, consider adding our `conda beta channel`_ alongside conda
-channels available for download in your condarc_ file.
-
-The location of ``.condarc`` is configurable, but it is often set in
-``${HOME}/.condarc``. Read about condarc_ online if you are in doubt.
-
-After our `conda beta channel`_ is included on your configuration, proceed as
-above to create an environment with the latest dependencies instead of the
-latest *stable* versions of each package.
-
-
-**Optional** Automated environment creation
-===========================================
-
-It is possible to automate the above procedure using a script that tries to
-automatically parse build and test requirements in ``conda/meta.yaml`` and
-create the required development environment from scratch. This script lives in
-the package ``bob.admin``. The first step is to checkout ``bob.admin``
-alongside the package you're developing:
-
-
-.. code-block:: sh
-
- $ cd ..
- $ ls
- <PACKAGE>
- $ git clone https://gitlab.idiap.ch/bob/bob.admin
- $ ls
- <PACKAGE>
- bob.admin
- $ cd <PACKAGE>
-
-
-.. note::
-
- If you already have checked out ``bob.admin``, make sure to update it with
- ``git pull``. We're constantly making improvements to this package.
-
-
-Once ``bob.admin`` is available alongside your package, make sure both
-``conda`` and ``conda-build`` are installed **and updated** on the base
-environment of your conda_ installation. The automated script requires conda_
-version 4.4 or above and ``conda-build`` version 3 or above. It also requires
-the package ``pyyaml`` to be installed on the base of your conda installation.
-Follow this recipe to get all up-to-date and ready:
-
-
-.. code-block:: sh
-
- $ conda update -n base conda conda-build
- ...
- $ conda install -n base pyyaml
- ...
-
-
-.. note::
-
- Notice the application ``conda`` is in my ``${PATH}`` and therefore the
- shell can find it easily. **Make sure you do the same**.
-
-
-Now that you're all set, just call the script
-``bob.admin/conda/conda-bootstrapy`` and pass the name of the resulting
-environment you'd like to create:
-
-
-.. code-block:: sh
-
- $ cd <PACKAGE>
- $ conda activate base
- $ ../bob.admin/conda/conda-bootstrap.py dev
- ...
-
-
-This will parse the conda recipe from your package and create a new conda
-environment on your conda installation called ``dev``. The environment ``dev``
-contains all build *and* test dependencies required for your package. Activate
-this environment and you're ready.
-
-.. note::
-
- By default, our script **will include** our `conda beta channel`_ while
- creating your environment. You may modify the file
- ``bob.admin/conda/build-condarc`` if you'd like to include or remove
- channels.
-
- In this setup, we bypass your own condarc_ setup and use a stock version
- provided with ``bob.admin``.
-
-
-Running buildout
-----------------
-
-The last step is to create a hooked-up environment so you can quickly test
-local changes to your package w/o necessarily creating a conda-package. This
-step is the easiest:
-
-
-.. code-block:: sh
-
- $ cd <PACKAGE> #if that is not the case
- $ conda activate dev
- $ buildout
- ...
-
-zc.buildout_ works by modifying the load paths of scripts to find the correct
-version of your package sources from the local checkout. After running,
-buildout creates a directory called ``bin`` on your local package checkout. Use
-the applications living there to develop your package. For example, if you need
-to run the test suite:
-
-
-.. code-block:: sh
-
- $ ./bin/nosetests -sv
-
-
-A python interpreter clone can be used to run interactive sessions:
-
-
-.. code-block:: sh
-
- $ ./bin/python
-
-
-**Optional** Cross-development of dependencies
-----------------------------------------------
-
-From time to time, you may wish to cross-develop multiple projects at once. For
-example, you may wish to develop ``bob.bio.face``, while *also* making
-modifications to ``bob.bio.base``. In this case, you'll need to create a
-buildout recipe (i.e., a new ``.cfg``) file that instructs buildout to also
-checkout the sources for ``bob.bio.base`` while setting up the local structure
-for ``bob.bio.face``. Follow our development guide from ``bob.extension`` at
-:ref:`bob.extension` for more instructions on this step. Once your new ``.cfg``
-is ready, use it like this setup:
-
-
-.. code-block:: sh
-
- $ buildout -c newrecipe.cfg
-
-
-.. include:: links.rst
diff --git a/doc/index.rst b/doc/index.rst
index 597f0936..94f44761 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -17,7 +17,6 @@ Documentation
install
release
- guide
development
additional
templates
--
GitLab