Move bob dev environment to this package

* Update ci to match bob.admin (use sed to change behaviour instead of cp)

.gitlab-ci.yml

@@ -28,7 +28,7 @@ stages:
     - ./_ci/install.sh _ci #updates
     - ./_ci/before_build.sh
   script:
-    - cp build.sh ./_ci  
+    - sed -ie 's/sphinx-build -W/sphinx-build/g' ./_ci/build.sh
     - ./_ci/build.sh
   after_script:
     - ./_ci/after_build.sh
@@ -48,7 +48,7 @@ stages:
     - ./_ci/install.sh _ci #updates
     - ./_ci/before_test.sh
   script:
-    - cp test.sh ./_ci  
+    - sed -ie '/doctest/d' ./_ci/test.sh
     - ./_ci/test.sh
   after_script:
     - ./_ci/after_test.sh

MANIFEST.in

-include README.rst
-include CHANGELOG
-include LICENSE
-include bootstrap-buildout.py
-include buildout.cfg
-include requirements.txt
+include README.rst CHANGELOG LICENSE buildout.cfg requirements.txt version.txt
 include test-requirements.txt
-include version.txt
 recursive-include doc doc/plot *.rst *.png conf.py *.py *.ico
-

README.rst

@@ -33,24 +33,26 @@ Installation
 ------------
 
 Follow our `installation`_ instructions. Then, using the Python interpreter
-provided by the distribution, bootstrap and buildout this package::
+provided by the distribution, build this package with::
+
+  $ buildout
 
-  $ python bootstrap-buildout.py
-  $ ./bin/buildout
-  
 
 For the maintainers
 -------------------
 
 In the next subsections we have instructions for the maintainers of the package.
 
+
 Adding a dependency package
 ===========================
 
-   
-   To add a package on bob, just append the package name in the file ('requirements.txt').
+
+To add a package on bob, just append the package name in the file
+(``requirements.txt``).
 
 .. warning::
+
    Before adding a package to this prototype, please ensure that the package:
 
    * contains a README clearly indicating how to install the package (including
@@ -70,11 +72,12 @@ Adding a dependency package
 Updating the dependencies
 =========================
 
- If you want to update the version of the dependency packages, run the following commands::
- 
- $ ./bin/python ./bob/script/get_versions.py > requirements.txt
- $ git commit requirements.txt -m "Update requeriments" && git push
- 
+If you want to update the version of the dependency packages, run the following
+commands::
+
+  $ ./bin/python ./bob/script/get_versions.py > requirements.txt
+  $ git commit requirements.txt -m "Update requeriments" && git push
+
 
 Removing a dependency package
 =============================
@@ -92,4 +95,3 @@ Removing a dependency package
 .. _bug tracker: https://www.idiap.ch/software/bob/issues
 .. _dependencies: https://gitlab.idiap.ch/bob/bob/wikis/Dependencies
 .. _installation: https://www.idiap.ch/software/bob/install
-

bootstrap-buildout.py

-##############################################################################
-#
-# Copyright (c) 2006 Zope Foundation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-"""Bootstrap a buildout-based project
-
-Simply run this script in a directory containing a buildout.cfg.
-The script accepts buildout command-line options, so you can
-use the -c option to specify an alternate configuration file.
-"""
-
-import os
-import shutil
-import sys
-import tempfile
-
-from optparse import OptionParser
-
-tmpeggs = tempfile.mkdtemp()
-
-usage = '''\
-[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
-
-Bootstraps a buildout-based project.
-
-Simply run this script in a directory containing a buildout.cfg, using the
-Python that you want bin/buildout to use.
-
-Note that by using --find-links to point to local resources, you can keep
-this script from going over the network.
-'''
-
-parser = OptionParser(usage=usage)
-parser.add_option("-v", "--version", help="use a specific zc.buildout version")
-
-parser.add_option("-t", "--accept-buildout-test-releases",
-                  dest='accept_buildout_test_releases',
-                  action="store_true", default=False,
-                  help=("Normally, if you do not specify a --version, the "
-                        "bootstrap script and buildout gets the newest "
-                        "*final* versions of zc.buildout and its recipes and "
-                        "extensions for you.  If you use this flag, "
-                        "bootstrap and buildout will get the newest releases "
-                        "even if they are alphas or betas."))
-parser.add_option("-c", "--config-file",
-                  help=("Specify the path to the buildout configuration "
-                        "file to be used."))
-parser.add_option("-f", "--find-links",
-                  help=("Specify a URL to search for buildout releases"))
-parser.add_option("--allow-site-packages",
-                  action="store_true", default=False,
-                  help=("Let bootstrap.py use existing site packages"))
-parser.add_option("--setuptools-version",
-                  help="use a specific setuptools version")
-
-
-options, args = parser.parse_args()
-
-######################################################################
-# load/install setuptools
-
-try:
-    if options.allow_site_packages:
-        import setuptools
-        import pkg_resources
-    from urllib.request import urlopen
-except ImportError:
-    from urllib2 import urlopen
-
-ez = {}
-exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez)
-
-if not options.allow_site_packages:
-    # ez_setup imports site, which adds site packages
-    # this will remove them from the path to ensure that incompatible versions
-    # of setuptools are not in the path
-    import site
-    # inside a virtualenv, there is no 'getsitepackages'.
-    # We can't remove these reliably
-    if hasattr(site, 'getsitepackages'):
-        for sitepackage_path in site.getsitepackages():
-            sys.path[:] = [x for x in sys.path if sitepackage_path not in x]
-
-setup_args = dict(to_dir=tmpeggs, download_delay=0)
-
-if options.setuptools_version is not None:
-    setup_args['version'] = options.setuptools_version
-
-ez['use_setuptools'](**setup_args)
-import setuptools
-import pkg_resources
-
-# This does not (always?) update the default working set.  We will
-# do it.
-for path in sys.path:
-    if path not in pkg_resources.working_set.entries:
-        pkg_resources.working_set.add_entry(path)
-
-######################################################################
-# Install buildout
-
-ws = pkg_resources.working_set
-
-cmd = [sys.executable, '-c',
-       'from setuptools.command.easy_install import main; main()',
-       '-mZqNxd', tmpeggs]
-
-find_links = os.environ.get(
-    'bootstrap-testing-find-links',
-    options.find_links or
-    ('http://downloads.buildout.org/'
-     if options.accept_buildout_test_releases else None)
-    )
-if find_links:
-    cmd.extend(['-f', find_links])
-
-setuptools_path = ws.find(
-    pkg_resources.Requirement.parse('setuptools')).location
-
-requirement = 'zc.buildout'
-version = options.version
-if version is None and not options.accept_buildout_test_releases:
-    # Figure out the most recent final version of zc.buildout.
-    import setuptools.package_index
-    _final_parts = '*final-', '*final'
-
-    def _final_version(parsed_version):
-        try:
-            return not parsed_version.is_prerelease
-        except AttributeError:
-            # Older setuptools
-            for part in parsed_version:
-                if (part[:1] == '*') and (part not in _final_parts):
-                    return False
-            return True
-
-    index = setuptools.package_index.PackageIndex(
-        search_path=[setuptools_path])
-    if find_links:
-        index.add_find_links((find_links,))
-    req = pkg_resources.Requirement.parse(requirement)
-    if index.obtain(req) is not None:
-        best = []
-        bestv = None
-        for dist in index[req.project_name]:
-            distv = dist.parsed_version
-            if _final_version(distv):
-                if bestv is None or distv > bestv:
-                    best = [dist]
-                    bestv = distv
-                elif distv == bestv:
-                    best.append(dist)
-        if best:
-            best.sort()
-            version = best[-1].version
-if version:
-    requirement = '=='.join((requirement, version))
-cmd.append(requirement)
-
-import subprocess
-if subprocess.call(cmd, env=dict(os.environ, PYTHONPATH=setuptools_path)) != 0:
-    raise Exception(
-        "Failed to execute command:\n%s" % repr(cmd)[1:-1])
-
-######################################################################
-# Import and run buildout
-
-ws.add_entry(tmpeggs)
-ws.require(requirement)
-import zc.buildout.buildout
-
-if not [a for a in args if '=' not in a]:
-    args.append('bootstrap')
-
-# if -c was provided, we push it back into args for buildout' main function
-if options.config_file is not None:
-    args[0:0] = ['-c', options.config_file]
-
-zc.buildout.buildout.main(args)
-shutil.rmtree(tmpeggs)

build.sh

-#!/usr/bin/env bash
-# Wed 21 Sep 2016 13:08:05 CEST
-
-source $(dirname ${0})/functions.sh
-
-run_cmd ./bin/buildout
-
-if [ -x ./bin/bob_dbmanage.py ]; then
-  run_cmd ./bin/bob_dbmanage.py all download --force;
-fi
-
-if [ -d ./doc ]; then
-  run_cmd ./bin/sphinx-build doc sphinx
-fi
-
-if [ -z "${WHEEL_TAG}" ]; then
-  # C/C++ extensions
-  run_cmd ./bin/python setup.py bdist_wheel
-else
-  # Python-only packages
-  run_cmd ./bin/python setup.py bdist_wheel --python-tag ${WHEEL_TAG}
-fi

doc/bob-devel-py27.yml

+name: bob-devel-py27
+channels:
+- https://www.idiap.ch/software/bob/conda
+- defaults
+dependencies:
+- python=2.7
+- cython
+- dask
+- hdf5
+- jpeg
+- libpng
+- libtiff
+- matplotlib
+- nose
+- numexpr
+- numpy
+- pillow
+- pip
+- psutil
+- scikit-image
+- scikit-learn
+- scipy
+- sphinx
+- sqlalchemy
+- anaconda=4.2.0 # [only linux]
+- anaconda=4.3.0 # [only osx]
+- boost=1.61
+- caffe # [only linux]
+- cmake
+- coverage
+- cyvlfeat=0.4.5
+- docopt
+- ffmpeg=2.8.10
+- gcc=4.8.5 # [only linux]
+- libgcc=4.8.5 # [only linux]
+- giflib=5.1.4
+- ipdb=0.10.2
+- ipython
+- libblitz=0.10
+- libmatio=1.5
+- libsvm=3.21
+- menpo=0.7.7
+- menpofit=0.4.1
+- menpowidgets=0.2.1p3
+- mne=0.13.1
+- openblas=0.2.19
+- opencv=3
+- pkgtools=0.7.3
+- pkg-config
+- pyedflib=0.1.6
+- schema=0.6.5
+- sox=14.4.2
+- sphinx_rtd_theme
+- toolchain=2.3.2
+- vlfeat=0.9.20
+- zc.buildout=2.9.2
+- pip:
+  - tensorflow==1.0.1

doc/bob-devel-py34.yml

+name: bob-devel-py34
+channels:
+- https://www.idiap.ch/software/bob/conda
+- defaults
+dependencies:
+- python=3.4
+- cython
+- dask
+- hdf5
+- jpeg
+- libpng
+- libtiff
+- matplotlib
+- nose
+- numexpr
+- numpy
+- pillow
+- pip
+- psutil
+- scikit-image
+- scikit-learn
+- scipy
+- sphinx
+- sqlalchemy
+- anaconda=4.2.0 # [only linux]
+- anaconda=4.3.0 # [only osx]
+- boost=1.61
+- caffe # [only linux]
+- cmake
+- coverage
+- cyvlfeat=0.4.5
+- docopt
+- ffmpeg=2.8.10
+- gcc=4.8.5 # [only linux]
+- libgcc=4.8.5 # [only linux]
+- giflib=5.1.4
+- ipdb=0.10.2
+- ipython
+- libblitz=0.10
+- libmatio=1.5
+- libsvm=3.21
+- menpo=0.7.7
+- menpofit=0.4.1
+- menpowidgets=0.2.1p3
+- mne=0.13.1
+- openblas=0.2.19
+- opencv=3
+- pkgtools=0.7.3
+- pkg-config
+- pyedflib=0.1.6
+- schema=0.6.5
+- sox=14.4.2
+- sphinx_rtd_theme
+- toolchain=2.3.2
+- vlfeat=0.9.20
+- zc.buildout=2.9.2
+- pip:
+  - tensorflow==1.0.1

doc/bob-devel-py35.yml

+name: bob-devel-py35
+channels:
+- https://www.idiap.ch/software/bob/conda
+- defaults
+dependencies:
+- python=3.5
+- cython
+- dask
+- hdf5
+- jpeg
+- libpng
+- libtiff
+- matplotlib
+- nose
+- numexpr
+- numpy
+- pillow
+- pip
+- psutil
+- scikit-image
+- scikit-learn
+- scipy
+- sphinx
+- sqlalchemy
+- anaconda=4.2.0 # [only linux]
+- anaconda=4.3.0 # [only osx]
+- boost=1.61
+- caffe # [only linux]
+- cmake
+- coverage
+- cyvlfeat=0.4.5
+- docopt
+- ffmpeg=2.8.10
+- gcc=4.8.5 # [only linux]
+- libgcc=4.8.5 # [only linux]
+- giflib=5.1.4
+- ipdb=0.10.2
+- ipython
+- libblitz=0.10
+- libmatio=1.5
+- libsvm=3.21
+- menpo=0.7.7
+- menpofit=0.4.1
+- menpowidgets=0.2.1p3
+- mne=0.13.1
+- openblas=0.2.19
+- opencv=3
+- pkgtools=0.7.3
+- pkg-config
+- pyedflib=0.1.6
+- schema=0.6.5
+- sox=14.4.2
+- sphinx_rtd_theme
+- toolchain=2.3.2
+- vlfeat=0.9.20
+- zc.buildout=2.9.2
+- pip:
+  - tensorflow==1.0.1

doc/bob-devel.yml

+bob-devel-py27.yml
\ No newline at end of file

doc/conf.py

@@ -41,14 +41,6 @@ except Exception:
 packages = get_dependencies()
 download_packages(packages[1:], temp_dir)
 
-# download a copy of from-scratch.sh from bob.admin
-url = "http://gitlab.idiap.ch/bob/bob.admin/raw/master/install/from-scratch.sh"
-try:
-  from urllib import urlretrieve
-except ImportError:
-  from urllib.request import urlretrieve
-urlretrieve(url, filename='from-scratch.sh')
-
 import sphinx
 if sphinx.__version__ >= "1.4.1":
     extensions.append('sphinx.ext.imgmath')

doc/install.rst

@@ -118,7 +118,7 @@ And now you are ready to run your programs.
    >>> import bob.io.base
    >>> ...
 
-.. note:: 
+.. note::
 
    Now that you have all the dependencies and required Bob packages installed,
    it is a good idea to follow our :doc:`temp/bob.extension/doc/buildout` page
@@ -330,9 +330,33 @@ be fetched automatically from PyPI otherwise.
 Setting up a Development Environment using Conda
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Please look at :download:`this script <from-scratch.sh>`
-that lets you setup a `conda`_ environment for |project| development and
-installs all the dependencies in one shot.
+Our :download:`conda environment file <bob-devel.yml>` lets you create a conda_
+environment from scratch for |project| development and installs all the
+dependencies in one shot. To create the default development environment,
+download ``bob-devel.yml`` from the link above and run the following conda_
+command:
+
+
+.. code-block:: sh
+
+   $ conda create env -f bob-devel.yml
+
+
+If you'd like to rename the environment, just do:
+
+
+.. code-block:: sh
+
+   $ conda create env -n mydev -f bob-devel.yml
+
+
+To install an environment using a different version of Python, first edit the
+file `bob-devel.yml` and change the line that says `python=2.7` to the variant
+you wish to install.
+
+
+Read ``conda create env --help`` for more information.
+
 
 
 Installing |project| from source
@@ -394,7 +418,7 @@ the following script can be used to install all core |project| packages using
    -------------------------
    #!/bin/bash
    set -e
-   
+
    get_layer() {
    case $1 in
      1)
@@ -423,7 +447,7 @@ the following script can be used to install all core |project| packages using
        ;;
    esac
    }
-   
+
    for layer in `seq 1 8`;
    do
      get_layer ${layer}

test.sh

-#!/usr/bin/env bash
-# Wed 21 Sep 2016 13:08:05 CEST
-
-source $(dirname ${0})/functions.sh
-
-run_cmd cd ${PREFIX}
-
-# The tests:
-run_cmd ${PREFIX}/bin/python ${BOB_PREFIX_PATH}/bin/coverage run --source=${CI_PROJECT_NAME} ${BOB_PREFIX_PATH}/bin/nosetests -sv ${CI_PROJECT_NAME}
-run_cmd ${PREFIX}/bin/python ${BOB_PREFIX_PATH}/bin/coverage report
-
-run_cmd cd ${CI_PROJECT_DIR}