Moving away from bootstrap-buildout.py

As you may have already noticed, setuptools is deprecating ez_setup.py, the module that allowed setuptools to install itself. With that, it has become evident that zc.buildout will no longer be able to self-bootstrap and, at a certain point, it will be removed. So far, a hack is put in place pinning the version of setuptools to 33.0.0 (it is now on 34.3.1 already...), which is the last to provide this functionality. See:

• https://github.com/buildout/buildout/issues/289
• https://github.com/pypa/setuptools/issues/581

The message printed out is:

$ python bootstrap-buildout.py
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools
Downloading https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip
...

I propose:

1. we start shipping zc.buildout as part of our bob-devel environment and as a bob dependence
2. we remove bootstrap-buildout.py from all packages
3. after activating the conda environment, the user does only buildout (instead of ./bin/buildout)
4. update instructions on bob.extension's guide to reflect these changes.

I tested the above changes and that seems to work fine. Are there any objections?

ping @bob

@andre.anjos If you know how to do that, please go ahead. I actually like it better to have only one step instead of two (bootstrap & bin/buildout). This makes it (a little) easier for people to install our packages.

I am also ok with this. Please do testing before changing the instructions.

zc.buildout is now installed on all CIs. bootstrapping is no longer required. We still need to modify the CI scripts to take that into account.

mentioned in commit 61c42e5e

CI scripts now don't attempt to bootstrap zc.buildout anymore.

mentioned in commit f24d5914

Templates and readme from this package are now updated to take this into account.

I think the instruction on the readme files should just be: for core bob packages:

This package is a **core** bob package. Please install bob or if you want to install just this package after you have added our channel:
conda install bob.package

for the rest:

This package is an **extra** or **extension** or **satellite** bob package. Please install bob first and then:
pip install bob.package

mentioned in commit dbd9705c

mentioned in issue beat/beat.web#462 (closed)

I think as a result of changes done because of this issue, our deployment step in the CI is broken. https://gitlab.idiap.ch/bob/bob.db.nist_sre12/builds/57702

Do we really need buildout? Looks like pip already does almost everything: https://pip.pypa.io/en/stable/user_guide/ Not to mention how slow buildout is: bob.buildout#19 (closed)

Ok, I think I have this figured out. We used to advertise buildout for both installation and development but now I think we can replace this by combination of conda and pip. Here are the scenarios:

• You can achieve the isolation that buildout provides for different experiments with conda environments.
  • conda create -n test_environment python=2 bob. This would also mean we move away from central conda installations too.
• Install a package: conda install <package>
• This cannot be easier than this. We will provide conda packages for all packages soon.
• Install a package from source into your python environment:
• conda install -n <env-name> gcc=4 libgcc=4 toolchain This line would be available in the documentation.
  • Having toolchain installed is important here. This sets some release mode compilation flags. Also some compilation flags to guarantee ABI compatibility with packages from our conda channel.
• source activate <env-name>
  • activate the environment to make sure the flags are set.
• pip install .
• Develop the current package:
  • Optionally: Export the debug flags:
    • export CFLAGS=$CFLAGS -O0 -g -DBOB_DEBUG -DBZ_DEBUG
    • export CXXFLAGS=$CXXFLAGS -O0 -g -DBOB_DEBUG -DBZ_DEBUG
  • pip install -e . the -e option means editable mode or develop mode.
• Develop the bob requirements too:

  • pip install -r requirements.txt while in your requirements.txt you have something like this (example for bob.io.image):

setuptools -egit+https://gitlab.idiap.ch/bob/bob.extension.git#egg=bob.extension -egit+https://gitlab.idiap.ch/bob/bob.blitz.git#egg=bob.blitz -egit+https://gitlab.idiap.ch/bob/bob.core.git#egg=bob.core -egit+https://gitlab.idiap.ch/bob/bob.io.base.git#egg=bob.io.base matplotlib

  * This will clone these requirements in `./src` and install them in develop mode.
  * Now there is a problem that this packages may get compiled a couple of times because they depend on each other. To avoid that you can do: 
    * `tr '\n' '\0' < requirements.txt | xargs -0 -n 1 pip install `

I think with a couple of changes in `bob.extension` and providing some scripts in `bob.extension` to ease the execution of commands above, we can totally ditch `buildout` and have faster development environments.
However, this need some training but I am willing to provide that if we decide to go this way.
@andre.anjos @pkorshunov @tiago.pereira @mguenther what do you think?

Further reading: https://pip.pypa.io/en/stable/user_guide/ and https://conda.io/docs/get-started.html

My only concern is that the new way does not make it harder to work compared to the old way. Currently, I need to run one command and will get packages needed for development inside ./src folder. I can modify these packages as I like, push back changes to git, if I want, and so on. Will this new approach allow me the same flexibility and ease of development? And what is the benefit? I'm also worried that we may stumble on some unexpected problems along the way.

At times, I feel that we are following HDD (hype driven development) paradigm (as was sent by @tiago.pereira https://blog.daftcode.pl/hype-driven-development-3469fc2e9b22) and it costs us too much time and effort.

My only concern is that the new way does not make it harder to work compared to the old way. Currently, I need to run one command and will get packages needed for development inside ./src folder. I can modify these packages as I like, push back changes to git, if I want, and so on. Will this new approach allow me the same flexibility and ease of development?

Yes, see: https://pip.pypa.io/en/stable/reference/pip_install/#git

And what is the benefit?

• We stop using an unsupported tool like buildout.
• We don't need to wait two minutes each time we run buildout.
• Our development will be more like the rest of Python developers using well tested and broadly used official tools.

I'm also worried that we may stumble on some unexpected problems along the way.

Probably but since we already use setuptools and pip for distribution the problems should be minimal. I have done some tests on different use cases and compared them to buildout. So far except doing a couple of extra commands and some required knowledge I have not seen a problem.

I don't know about hype or not but buildout seems like a monkey patched version of setuptools which does not seem to see much of development anymore. However pip is the official way of doing things which builds upon setuptools in a proper way and their development goes hand to hand under one organization: https://github.com/pypa/

Also see the official python package development guide: https://packaging.python.org/

I was just reviewing this - I'm in favor of exploring ways to avoid buildout. Still, it is not unmaintained - it is just not going to the same place we are. It is a tool for deploying pure-Python packages.

Pip also have problems:

1. As you have already noticed, you can't pip install all packages at once as we can with buildout (provided bob.buildout is used on the extension list)
2. We'll be replacing one single entry point by multiple different commands - or see 3.
3. The solution using bob.extension may be good, but we have to investigate it with care. A solution with a package that does not depend on Bob may be easier to use. We'd need to mimick this bootstrap/buildout at a certain point though. The starting point is having miniconda installed somewhere.

The minimum workflow could be the following:

0. User pre-installs miniconda somewhere (user maintains this, so it is not really part of the everyday procedure)
1. User downloads a script from somewhere (maybe bob.extension)
2. User executes said script with a "requirements.txt" or similar file as you examplified - after that, the environment is ready for use

Another option:

0. Same as before
1. All users include a copy of the bootstrap script with their package
2. To get the environment ready, user runs the bootstrap script against requirements.txt or similar as you examplified

mentioned in issue #47 (closed)

User downloads a script from somewhere (maybe bob.extension)

What should this script do? Why can't we just trust that if someone really wants to develop something they should at least know about it a little?

About the development aspect; honestly, aside from Bob and its satellites, I've never collaborated to projects where I needed to constantly patch the dependencies too. Most of the time I'm working with 2 or 3 dependencies.

This feature from zc.buildout is just great. In one shot pull everything, either from git, from a certain branch from git, something in your file system. Would be good to do this with pip (I know this feature https://pip.pypa.io/en/stable/reference/pip_install/#git).

I'm kinda agree with @amohammadi, if you are developing you should know a little about it, but if you can make it simpler this would destroy one barrier in our development cycle.

I think all theses problems stem from the fact that instead of putting these things in documentation we just create magical scripts that sets up development environment for people. This puts pressure on us maintaining these scripts. Whenever for some reason somebody's buildout breaks is because they are not trained enough and they keep blaming it on us who know and have actually put time and investigated this. If the developer doesn't know about these things maybe they shouldn't be developing these packages in the first place.

Examples: bob.buildout#18 (closed) bob.buildout#17 (closed)

I do not think we should be dropping support for buildout. Instead we can have two development guides in bob.extension. One using pip and one using buildout. People (developers) should be free to use whatever they want as long as they know what they are doing.

So my conclusion is to remove mentions of buildout from documentation: bob#241 (closed) and assume that when people want to use (not develop) something, they have installed the package with either conda or pip. Then just have one or two developer guides only in one place that explains the peculiarities of buildout or pip or whatever.

I mean how hard is it to tell people that you need to export debug flags if you want to compile in debug mode. Or that you need to compile theses packages one by one because pip can't handle it.

Developing with pip is bound to work out of the box, so why not adding a section on the manual? Looks like a good compromise for the time being. We should keep an eye on feedback after this.

As for the script, only experience will tell. My gut feeling at this point is that remembering command-lines is hard and people tend to code this in scripts. Each will have one and we'll end-up debugging those rather than buildout or our own script. Not having the script does not necessarily mean it is going to be easier to handle people's requests, unfortunately. It is a complex problem.

mentioned in merge request bob.extension!37 (merged)

changed milestone to %May 2017 Hackathon

added Easy label

Can we close this one now?

not until we merge the python36 branches. @akomaty I don't see these branches on all bob packages. Dit the script fail? For example bob.db.base

changed milestone to %Bob 2.7.x release

There are no branches anywhere. If you want to get on with this, please push that. Assign milestone "Bob 2.7.x release" to each MR, please.

Relevant branches for py36 are now merged. The bootstrap files have been removed from everywhere. Can we close it?

assigned to @amohammadi

closed

Fixed in most packages.