Commit 7e074359 authored by André Anjos's avatar André Anjos 💬
Browse files

Move bob dev environment to this package

* Update ci to match bob.admin (use sed to change behaviour instead of cp)
parent 117de8a6
Pipeline #7765 failed with stages
in 22 minutes and 49 seconds
......@@ -28,7 +28,7 @@ stages:
- ./_ci/install.sh _ci #updates
- ./_ci/before_build.sh
script:
- cp build.sh ./_ci
- sed -ie 's/sphinx-build -W/sphinx-build/g' ./_ci/build.sh
- ./_ci/build.sh
after_script:
- ./_ci/after_build.sh
......@@ -48,7 +48,7 @@ stages:
- ./_ci/install.sh _ci #updates
- ./_ci/before_test.sh
script:
- cp test.sh ./_ci
- sed -ie '/doctest/d' ./_ci/test.sh
- ./_ci/test.sh
after_script:
- ./_ci/after_test.sh
......
include README.rst
include CHANGELOG
include LICENSE
include bootstrap-buildout.py
include buildout.cfg
include requirements.txt
include README.rst CHANGELOG LICENSE buildout.cfg requirements.txt version.txt
include test-requirements.txt
include version.txt
recursive-include doc doc/plot *.rst *.png conf.py *.py *.ico
......@@ -33,24 +33,26 @@ Installation
------------
Follow our `installation`_ instructions. Then, using the Python interpreter
provided by the distribution, bootstrap and buildout this package::
provided by the distribution, build this package with::
$ buildout
$ python bootstrap-buildout.py
$ ./bin/buildout
For the maintainers
-------------------
In the next subsections we have instructions for the maintainers of the package.
Adding a dependency package
===========================
To add a package on bob, just append the package name in the file ('requirements.txt').
To add a package on bob, just append the package name in the file
(``requirements.txt``).
.. warning::
Before adding a package to this prototype, please ensure that the package:
* contains a README clearly indicating how to install the package (including
......@@ -70,11 +72,12 @@ Adding a dependency package
Updating the dependencies
=========================
If you want to update the version of the dependency packages, run the following commands::
$ ./bin/python ./bob/script/get_versions.py > requirements.txt
$ git commit requirements.txt -m "Update requeriments" && git push
If you want to update the version of the dependency packages, run the following
commands::
$ ./bin/python ./bob/script/get_versions.py > requirements.txt
$ git commit requirements.txt -m "Update requeriments" && git push
Removing a dependency package
=============================
......@@ -92,4 +95,3 @@ Removing a dependency package
.. _bug tracker: https://www.idiap.ch/software/bob/issues
.. _dependencies: https://gitlab.idiap.ch/bob/bob/wikis/Dependencies
.. _installation: https://www.idiap.ch/software/bob/install
##############################################################################
#
# Copyright (c) 2006 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout-based project
Simply run this script in a directory containing a buildout.cfg.
The script accepts buildout command-line options, so you can
use the -c option to specify an alternate configuration file.
"""
import os
import shutil
import sys
import tempfile
from optparse import OptionParser
tmpeggs = tempfile.mkdtemp()
usage = '''\
[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
Bootstraps a buildout-based project.
Simply run this script in a directory containing a buildout.cfg, using the
Python that you want bin/buildout to use.
Note that by using --find-links to point to local resources, you can keep
this script from going over the network.
'''
parser = OptionParser(usage=usage)
parser.add_option("-v", "--version", help="use a specific zc.buildout version")
parser.add_option("-t", "--accept-buildout-test-releases",
dest='accept_buildout_test_releases',
action="store_true", default=False,
help=("Normally, if you do not specify a --version, the "
"bootstrap script and buildout gets the newest "
"*final* versions of zc.buildout and its recipes and "
"extensions for you. If you use this flag, "
"bootstrap and buildout will get the newest releases "
"even if they are alphas or betas."))
parser.add_option("-c", "--config-file",
help=("Specify the path to the buildout configuration "
"file to be used."))
parser.add_option("-f", "--find-links",
help=("Specify a URL to search for buildout releases"))
parser.add_option("--allow-site-packages",
action="store_true", default=False,
help=("Let bootstrap.py use existing site packages"))
parser.add_option("--setuptools-version",
help="use a specific setuptools version")
options, args = parser.parse_args()
######################################################################
# load/install setuptools
try:
if options.allow_site_packages:
import setuptools
import pkg_resources
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
ez = {}
exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez)
if not options.allow_site_packages:
# ez_setup imports site, which adds site packages
# this will remove them from the path to ensure that incompatible versions
# of setuptools are not in the path
import site
# inside a virtualenv, there is no 'getsitepackages'.
# We can't remove these reliably
if hasattr(site, 'getsitepackages'):
for sitepackage_path in site.getsitepackages():
sys.path[:] = [x for x in sys.path if sitepackage_path not in x]
setup_args = dict(to_dir=tmpeggs, download_delay=0)
if options.setuptools_version is not None:
setup_args['version'] = options.setuptools_version
ez['use_setuptools'](**setup_args)
import setuptools
import pkg_resources
# This does not (always?) update the default working set. We will
# do it.
for path in sys.path:
if path not in pkg_resources.working_set.entries:
pkg_resources.working_set.add_entry(path)
######################################################################
# Install buildout
ws = pkg_resources.working_set
cmd = [sys.executable, '-c',
'from setuptools.command.easy_install import main; main()',
'-mZqNxd', tmpeggs]
find_links = os.environ.get(
'bootstrap-testing-find-links',
options.find_links or
('http://downloads.buildout.org/'
if options.accept_buildout_test_releases else None)
)
if find_links:
cmd.extend(['-f', find_links])
setuptools_path = ws.find(
pkg_resources.Requirement.parse('setuptools')).location
requirement = 'zc.buildout'
version = options.version
if version is None and not options.accept_buildout_test_releases:
# Figure out the most recent final version of zc.buildout.
import setuptools.package_index
_final_parts = '*final-', '*final'
def _final_version(parsed_version):
try:
return not parsed_version.is_prerelease
except AttributeError:
# Older setuptools
for part in parsed_version:
if (part[:1] == '*') and (part not in _final_parts):
return False
return True
index = setuptools.package_index.PackageIndex(
search_path=[setuptools_path])
if find_links:
index.add_find_links((find_links,))
req = pkg_resources.Requirement.parse(requirement)
if index.obtain(req) is not None:
best = []
bestv = None
for dist in index[req.project_name]:
distv = dist.parsed_version
if _final_version(distv):
if bestv is None or distv > bestv:
best = [dist]
bestv = distv
elif distv == bestv:
best.append(dist)
if best:
best.sort()
version = best[-1].version
if version:
requirement = '=='.join((requirement, version))
cmd.append(requirement)
import subprocess
if subprocess.call(cmd, env=dict(os.environ, PYTHONPATH=setuptools_path)) != 0:
raise Exception(
"Failed to execute command:\n%s" % repr(cmd)[1:-1])
######################################################################
# Import and run buildout
ws.add_entry(tmpeggs)
ws.require(requirement)
import zc.buildout.buildout
if not [a for a in args if '=' not in a]:
args.append('bootstrap')
# if -c was provided, we push it back into args for buildout' main function
if options.config_file is not None:
args[0:0] = ['-c', options.config_file]
zc.buildout.buildout.main(args)
shutil.rmtree(tmpeggs)
#!/usr/bin/env bash
# Wed 21 Sep 2016 13:08:05 CEST
source $(dirname ${0})/functions.sh
run_cmd ./bin/buildout
if [ -x ./bin/bob_dbmanage.py ]; then
run_cmd ./bin/bob_dbmanage.py all download --force;
fi
if [ -d ./doc ]; then
run_cmd ./bin/sphinx-build doc sphinx
fi
if [ -z "${WHEEL_TAG}" ]; then
# C/C++ extensions
run_cmd ./bin/python setup.py bdist_wheel
else
# Python-only packages
run_cmd ./bin/python setup.py bdist_wheel --python-tag ${WHEEL_TAG}
fi
name: bob-devel-py27
channels:
- https://www.idiap.ch/software/bob/conda
- defaults
dependencies:
- python=2.7
- cython
- dask
- hdf5
- jpeg
- libpng
- libtiff
- matplotlib
- nose
- numexpr
- numpy
- pillow
- pip
- psutil
- scikit-image
- scikit-learn
- scipy
- sphinx
- sqlalchemy
- anaconda=4.2.0 # [only linux]
- anaconda=4.3.0 # [only osx]
- boost=1.61
- caffe # [only linux]
- cmake
- coverage
- cyvlfeat=0.4.5
- docopt
- ffmpeg=2.8.10
- gcc=4.8.5 # [only linux]
- libgcc=4.8.5 # [only linux]
- giflib=5.1.4
- ipdb=0.10.2
- ipython
- libblitz=0.10
- libmatio=1.5
- libsvm=3.21
- menpo=0.7.7
- menpofit=0.4.1
- menpowidgets=0.2.1p3
- mne=0.13.1
- openblas=0.2.19
- opencv=3
- pkgtools=0.7.3
- pkg-config
- pyedflib=0.1.6
- schema=0.6.5
- sox=14.4.2
- sphinx_rtd_theme
- toolchain=2.3.2
- vlfeat=0.9.20
- zc.buildout=2.9.2
- pip:
- tensorflow==1.0.1
name: bob-devel-py34
channels:
- https://www.idiap.ch/software/bob/conda
- defaults
dependencies:
- python=3.4
- cython
- dask
- hdf5
- jpeg
- libpng
- libtiff
- matplotlib
- nose
- numexpr
- numpy
- pillow
- pip
- psutil
- scikit-image
- scikit-learn
- scipy
- sphinx
- sqlalchemy
- anaconda=4.2.0 # [only linux]
- anaconda=4.3.0 # [only osx]
- boost=1.61
- caffe # [only linux]
- cmake
- coverage
- cyvlfeat=0.4.5
- docopt
- ffmpeg=2.8.10
- gcc=4.8.5 # [only linux]
- libgcc=4.8.5 # [only linux]
- giflib=5.1.4
- ipdb=0.10.2
- ipython
- libblitz=0.10
- libmatio=1.5
- libsvm=3.21
- menpo=0.7.7
- menpofit=0.4.1
- menpowidgets=0.2.1p3
- mne=0.13.1
- openblas=0.2.19
- opencv=3
- pkgtools=0.7.3
- pkg-config
- pyedflib=0.1.6
- schema=0.6.5
- sox=14.4.2
- sphinx_rtd_theme
- toolchain=2.3.2
- vlfeat=0.9.20
- zc.buildout=2.9.2
- pip:
- tensorflow==1.0.1
name: bob-devel-py35
channels:
- https://www.idiap.ch/software/bob/conda
- defaults
dependencies:
- python=3.5
- cython
- dask
- hdf5
- jpeg
- libpng
- libtiff
- matplotlib
- nose
- numexpr
- numpy
- pillow
- pip
- psutil
- scikit-image
- scikit-learn
- scipy
- sphinx
- sqlalchemy
- anaconda=4.2.0 # [only linux]
- anaconda=4.3.0 # [only osx]
- boost=1.61
- caffe # [only linux]
- cmake
- coverage
- cyvlfeat=0.4.5
- docopt
- ffmpeg=2.8.10
- gcc=4.8.5 # [only linux]
- libgcc=4.8.5 # [only linux]
- giflib=5.1.4
- ipdb=0.10.2
- ipython
- libblitz=0.10
- libmatio=1.5
- libsvm=3.21
- menpo=0.7.7
- menpofit=0.4.1
- menpowidgets=0.2.1p3
- mne=0.13.1
- openblas=0.2.19
- opencv=3
- pkgtools=0.7.3
- pkg-config
- pyedflib=0.1.6
- schema=0.6.5
- sox=14.4.2
- sphinx_rtd_theme
- toolchain=2.3.2
- vlfeat=0.9.20
- zc.buildout=2.9.2
- pip:
- tensorflow==1.0.1
bob-devel-py27.yml
\ No newline at end of file
......@@ -41,14 +41,6 @@ except Exception:
packages = get_dependencies()
download_packages(packages[1:], temp_dir)
# download a copy of from-scratch.sh from bob.admin
url = "http://gitlab.idiap.ch/bob/bob.admin/raw/master/install/from-scratch.sh"
try:
from urllib import urlretrieve
except ImportError:
from urllib.request import urlretrieve
urlretrieve(url, filename='from-scratch.sh')
import sphinx
if sphinx.__version__ >= "1.4.1":
extensions.append('sphinx.ext.imgmath')
......
......@@ -118,7 +118,7 @@ And now you are ready to run your programs.
>>> import bob.io.base
>>> ...
.. note::
.. note::
Now that you have all the dependencies and required Bob packages installed,
it is a good idea to follow our :doc:`temp/bob.extension/doc/buildout` page
......@@ -330,9 +330,33 @@ be fetched automatically from PyPI otherwise.
Setting up a Development Environment using Conda
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please look at :download:`this script <from-scratch.sh>`
that lets you setup a `conda`_ environment for |project| development and
installs all the dependencies in one shot.
Our :download:`conda environment file <bob-devel.yml>` lets you create a conda_
environment from scratch for |project| development and installs all the
dependencies in one shot. To create the default development environment,
download ``bob-devel.yml`` from the link above and run the following conda_
command:
.. code-block:: sh
$ conda create env -f bob-devel.yml
If you'd like to rename the environment, just do:
.. code-block:: sh
$ conda create env -n mydev -f bob-devel.yml
To install an environment using a different version of Python, first edit the
file `bob-devel.yml` and change the line that says `python=2.7` to the variant
you wish to install.
Read ``conda create env --help`` for more information.
Installing |project| from source
......@@ -394,7 +418,7 @@ the following script can be used to install all core |project| packages using
-------------------------
#!/bin/bash
set -e
get_layer() {
case $1 in
1)
......@@ -423,7 +447,7 @@ the following script can be used to install all core |project| packages using
;;
esac
}
for layer in `seq 1 8`;
do
get_layer ${layer}
......
#!/usr/bin/env bash
# Wed 21 Sep 2016 13:08:05 CEST
source $(dirname ${0})/functions.sh
run_cmd cd ${PREFIX}
# The tests:
run_cmd ${PREFIX}/bin/python ${BOB_PREFIX_PATH}/bin/coverage run --source=${CI_PROJECT_NAME} ${BOB_PREFIX_PATH}/bin/nosetests -sv ${CI_PROJECT_NAME}
run_cmd ${PREFIX}/bin/python ${BOB_PREFIX_PATH}/bin/coverage report
run_cmd cd ${CI_PROJECT_DIR}
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment