Commit 6edeb46c authored by Manuel Günther's avatar Manuel Günther

Updated documentation; added 'build' step to new_version.py

parent b257c2ef
......@@ -63,6 +63,9 @@ Third, we have to add an extension using the ``Extension`` class, by listing all
.. code-block:: python
# read version from version.txt file
version = open("version.txt").read().rstrip()
setup(
...
setup_requires = build_requires,
......@@ -76,7 +79,7 @@ Third, we have to add an extension using the ``Extension`` class, by listing all
# the Python bindings
"bob/example/extension/main.cpp",
],
version = "1.0.0",
version = version,
bob_packages = bob_packages
),
... #add more extensions if you wish
......@@ -191,7 +194,7 @@ To generate a Library, simply add it in the list of ``ext_modules``:
[
"bob/example/library/cpp/Function.cpp",
],
version = "1.0.0",
version = version,
bob_packages = bob_packages,
),
# all other extensions will automatically link against the Library defined above
......@@ -201,7 +204,7 @@ To generate a Library, simply add it in the list of ``ext_modules``:
# the Python bindings
"bob/example/library/main.cpp",
],
version = "1.0.0",
version = version,
bob_packages = bob_packages,
),
... #add more Extensions if you wish
......
......@@ -52,6 +52,7 @@ The anatomy of a minimal package should look like the following:
+-- bootstrap.py # stock script downloaded from zc.buildout's website
+-- buildout.cfg # buildout configuration
+-- setup.py # installation + requirements for this particular package
+-- version.txt # the (current) version of your package
+-- doc # documentation directory
| +-- conf.py # Sphinx configuration
| +-- index.rst # Documentation starting point for Sphinx
......@@ -101,7 +102,7 @@ Particularly, it inspects the ``setup.py`` file in the root directory of the pac
setup(
name = 'bob.example.project',
version = '0.0.1a0',
version = open("version.txt").read().rstrip(),
...
packages = find_packages(),
...
......@@ -180,7 +181,7 @@ Here is how to go from nothing to everything:
.. code-block:: sh
$ /idiap/group/torch5spro/nightlies/last/bob.python/linux-x86_64-release/bin/python bootstrap.py
$ /idiap/group/torch5spro/nightlies/last/bob/linux-x86_64-release/bin/python bootstrap.py
.. warning::
This python version is replaced every night.
......@@ -190,14 +191,13 @@ Here is how to go from nothing to everything:
Change the directory to the installed directory of the latest stable packages at Idiap, as soon as they are published.
Also, as long as the packages are not uploaded to PyPI_, running ``./bin/buildout`` will not find the packages (yet).
Use:
Sometimes, you don't want to use the packages that are published on PyPI_, but the latest (nightly compiled) versions of all our packages.
In this case, use:
.. code-block:: sh
$ ./bin/buildout buildout:find-links=https://www.idiap.ch/software/bob/packages/bob.python/nightlies/last buildout:prefer-final=false
$ ./bin/buildout buildout:find-links=https://www.idiap.ch/software/bob/packages/nightlies/last buildout:prefer-final=false
to get the latest (nightly compiled) versions of all our packages (particularly of ``bob.buildout``).
......@@ -270,7 +270,7 @@ Documentation
=============
To write documentation, use the `Sphinx`_ Documentation Generator.
A template has been setup for you under the ``docs`` directory.
A template has been setup for you under the ``doc`` directory.
Get familiar with Sphinx and then unleash the writer in you.
Once you have edited both ``doc/conf.py`` and ``doc/index.rst`` you can run the documentation generator executing:
......@@ -290,7 +290,7 @@ You can find more options for ``sphinx-build`` using the ``-h`` flag:
.. note::
If the code you are distributing corresponds to the work described in a publication, don't forget to mention it in your ``README.rst`` file.
If the code you are distributing corresponds to the work described in a publication, don't forget to mention it in your ``doc/index.rst`` file.
Unit Tests
......@@ -314,9 +314,17 @@ To run the test units on your package call:
Distributing Your Work
----------------------
To distribute a package, we recommend you use PyPI.
To distribute a package, we recommend you use PyPI_.
`The Hitchhiker’s Guide to Packaging <http://guide.python-distribute.org/>`_ contains details and good examples on how to achieve this.
Particularly, you should edit your ``README.rst`` file to have a proper description of your package.
This file will be used to generate the front page of your package on PyPI_ and will, hence, be the first contact point of the world with your package.
.. note::
If you are writing a package to extend Bob, you might want to follow the README structure of all Bob packages.
The ``README.rst`` of **this package** (``bob.extension``) is a good example, including all the badges that show the current status of the package and the link to relevant information.
To ease up your life, we also provide a script to run all steps to publish your package.
Please read the following paragraphs to understand the steps in the ``./scripts/new_version.py`` script that will be explained at the end of this section.
Version Numbering Scheme
========================
......@@ -387,11 +395,13 @@ Here is a set of steps we recommend you follow when releasing a new version of y
The ``-``'s mark commits on your repository.
Time flies from left to right.
In this ficticious representation, the ``master`` branch continue under development, but one can see older branches don't receive much attention anymore.
In this fictitious representation, the ``master`` branch continue under development, but one can see older branches don't receive much attention anymore.
Here is an example for creating a branch at github (many of our satellite packages are hosted there).
Let's create a branch called ``1.1``::
.. code-block:: sh
$ git branch 1.1
$ git checkout 1.1
$ git push origin 1.1
......@@ -399,6 +409,8 @@ Here is a set of steps we recommend you follow when releasing a new version of y
3. When you decide to release something publicly, we recommend you **tag** the version of the package on your repository, so you have a marker to what code you actually published on PyPI.
Tagging on github would go like this::
.. code-block:: sh
$ git tag v1.1.0
$ git push && git push --tags
......@@ -407,19 +419,23 @@ Here is a set of steps we recommend you follow when releasing a new version of y
4. Finally, after branching and tagging, it is time for you to publish your new package on PyPI.
When the package is ready and you have tested it, just do the following::
$ python setup.py register #if you modified your setup.py or README.rst
$ python setup.py sdist --formats zip upload
.. code-block:: sh
$ ./bin/python setup.py register #if you modified your setup.py or README.rst
$ ./bin/python setup.py sdist --formats zip upload
.. note::
You can also check the .zip file that will be uploaded to PyPI before actually uploading it.
Just call::
Just call:
.. code-block:: sh
$ python setup.py sdist --formats zip
$ ./bin/python setup.py sdist --formats zip
and check what was put into the ``dist`` directory.
.. note::
To be able to upload a package to PyPI_ you have to register at the web page using a username and password.
To be able to upload a package to PyPI_ you have to register at the web page using a user name and password.
5. Announce the update on the relevant channels.
......@@ -430,28 +446,42 @@ Upload Additional Documentation to PythonHosted.org
In case you have written additional sphinx documentation in your satellite package that you want to share with the world, there is an easy way to push the documentation to `PythonHosted.org <http://pythonhosted.org>`_.
More detailed information are given `here <http://pythonhosted.org/an_example_pypi_project/buildanduploadsphinx.html>`__, which translates roughly into:
1. Edit your ``buildout.cfg`` and add the required package ``sphinx-pypi-upload``:
.. code-block:: sh
$ ./bin/python setup.py build_sphinx --source-dir=doc --build-dir=build/doc --all-files
$ ./bin/python setup.py upload_docs --upload-dir=build/doc/html
.. code-block:: python
The link to the documentation will automatically be added to the PyPI page of your package.
Usually it is a good idea to check the documentation after building and before uploading.
eggs = ...
sphinx-pypi-upload
And re-run ``buildout``::
Change the Version of your Satellite Package
============================================
$ ./bin/buildout
It is well understood that it requires quite some work to understand and follow the steps to publish (a new version) of your package.
Especially, when you want to update the .git repository and the version on PyPI_ at the same time.
In total, 5 steps need to be performed, in the right order.
These steps are:
.. note::
When you are using our ``bob.buildout:scripts`` recipe in your ``buildout.cfg`` (the default), you can skip this step since ``sphinx-pypi-upload`` is enabled by default.
1. Adding a tag in your git repository, possibly after changing the version of your package.
2. Running buildout to build your package.
3. Register and upload your package at PyPI.
4. Upload the documentation of your package to PythonHosted.org.
and, finally, to keep track of new changes:
2. Create and upload the documentation, following the instructions `here <https://pypi.python.org/pypi/Sphinx-PyPI-upload>`__::
5. Switch to a new version number.
$ ./bin/python setup.py build_sphinx --source-dir=doc --build-dir=build/doc --all-files
$ ./bin/python setup.py upload_docs --upload-dir=build/doc/html
All these steps are combined in the ``./scripts/new_version.py`` script.
This script needs to be run from within the root directory of your package.
Please run:
.. code-block:: sh
$ [PATH/TO/]bob.extension/script/new_version.py --help
to see a list of options.
The link to the documentation will automatically be added to the PyPI page of your package.
Usually it is a good idea to check the documentation after building and before uploading.
Satellite Packages Available
......
......@@ -8,24 +8,33 @@ import argparse
from distutils.version import StrictVersion as Version
doc = """
This script will push a new version of the given package on GitHub and PyPI.
It assumes that you are in the main directory of the package and have successfully ran buildout, and that you have submitted all changes that should go into the new version.
By default, four steps are executed, in this order:
- tag: The --stable-version will be set, added to GitHub, tagged in GitHub and pushed.
- pypi: The --stable-version will be registered and uploaded to PyPI
This script will push a new 'stable' version of the current package on GitHub and PyPI, and update the new version of the package to the given 'latest' version.
It assumes that you are in the main directory of the package and have successfully ran bootstrap, and that you have submitted all changes that should go into the new version.
For database packages, it also assumes that the .sql3 files have been generated.
Also, it assumes that the 'stable' version has not yet been uploaded to PyPI, and that no GitHub tag with this version exists.
The 'stable' version (i.e., what will be downloadable from PyPI) can be current version of the package, but not lower than that.
The 'latest' version (i.e., what will be the new master branch on GitHub) must be higher than the current and than the stable version.
By default, five steps are executed, in this order:
- tag: If given, the --stable-version will be set and added to GitHub; and the version is tagged in GitHub and pushed.
- build: The package will be (re-)built with bin/buildout using the provided build options.
- pypi: The --stable-version (or the current version) will be registered and uploaded to PyPI
- docs: The documentation will be generated and uploaded to PythonHosted
- latest: The --latest-version will be set and committed to GitHub
If any of these commands fail, the remaining steps will be skipped, unless you specify the --keep-going option
If any of these commands fail, the remaining steps will be skipped, unless you specify the --keep-going option.
If you only want a subset of the steps to be executed, you can limit them using the --steps option.
A valid use case, e.g., is only to re-upload the documentation.
"""
parser = argparse.ArgumentParser(description=doc, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--stable-version", '-s', required = True, help = "The stable version for the package")
parser.add_argument("--latest-version", '-l', required = True, help = "The latest version for the package")
parser.add_argument("--steps", nargs = "+", choices = ['tag', 'pypi', 'docs', 'latest'], default = ['tag', 'pypi', 'docs', 'latest'], help = "Select the steps that you want to execute")
parser.add_argument("--stable-version", '-s', help = "The stable version for the package; if not given, the current version is used")
parser.add_argument("--build-options", '-b', nargs='+', default = [], help = "Add options to build your package")
parser.add_argument("--steps", nargs = "+", choices = ['tag', 'build', 'pypi', 'docs', 'latest'], default = ['tag', 'build', 'pypi', 'docs', 'latest'], help = "Select the steps that you want to execute")
parser.add_argument("--dry-run", '-q', action = 'store_true', help = "Only print the actions, but do not execute them")
parser.add_argument("--keep-going", '-f', action = 'store_true', help = "Run all steps, even if some of them fail. HANDLE THIS FLAG WITH CARE!")
parser.add_argument("--verbose", '-v', action = 'store_true', help = "Print more information")
......@@ -35,13 +44,15 @@ args = parser.parse_args()
# assert the the version file is there
version_file = 'version.txt'
assert os.path.exists(version_file)
if not os.path.exists(version_file):
print("Could not find the file '%s' containing the version number. Are you inside the root directory of your package?")
sys.exit(-1)
def run_commands(version, *calls):
"""Updates the version.txt to the given version and runs the given commands."""
if args.verbose or args.dry_run:
if version is not None and (args.verbose or args.dry_run):
print (" - cat '%s' > %s" % (version, version_file))
if not args.dry_run:
if not args.dry_run and version is not None:
# update version to stable version, if not done yet
with open(version_file, 'w') as f:
f.write(version)
......@@ -56,45 +67,54 @@ def run_commands(version, *calls):
# call failed (has non-zero exit status)
print ("Command '%s' failed; stopping" % ' '.join(call))
if not args.keep_going:
exit(-1)
sys.exit(-1)
# get current version
current_version = open(version_file).read().rstrip()
# check the versions
if Version(args.latest_version) <= Version(args.stable_version):
if args.stable_version is not None and Version(args.latest_version) <= Version(args.stable_version):
print ("The latest version '%s' must be greater than the stable version '%s'" % (args.latest_version, args.stable_version))
sys.exit(-1)
if Version(current_version) >= Version(args.latest_version):
print ("The latest version '%s' must be greater than the current version '%s'" % (args.latest_version, current_version))
sys.exit(-1)
if args.stable_version is not None and Version(current_version) > Version(args.stable_version):
print ("The stable version '%s' cannot be smaller than the current version '%s'" % (args.stable_version, current_version))
sys.exit(-1)
if 'tag' in args.steps:
if Version(current_version) != Version(args.stable_version):
print ("\nTagging version '%s'" % args.stable_version)
# update stable version on github and add a tag
run_commands(args.stable_version, ['git', 'add', 'version.txt'], ['git', 'commit', '-m', '"Increased version to %s [skip ci]"' % args.stable_version], ['git', 'tag', 'v%s' % args.stable_version], ['git', 'push', '--tags'])
if args.stable_version is not None and Version(args.stable_version) > Version(current_version):
# update stable version on github
run_commands(args.stable_version, ['git', 'add', 'version.txt'], ['git', 'commit', '-m', 'Increased version to %s [skip ci]' % args.stable_version])
else:
print ("\nSkipping the 'tag' step since the the current version '%s' is already the stable version '%s'" % (current_version, args.stable_version))
# assure that we have the current version
args.stable_version = current_version
# add a github tag
print ("\nTagging version '%s'" % args.stable_version)
run_commands(None, ['git', 'tag', 'v%s' % args.stable_version], ['git', 'push', '--tags'])
if 'build' in args.steps:
print ("\nBuilding the package")
run_commands(None, ['./bin/buildout'] + args.build_options)
if 'pypi' in args.steps:
if Version(current_version) != Version(args.stable_version):
print ("\nUploading version '%s' to PyPI" % args.stable_version)
# update version on github and add a tag
run_commands(args.stable_version, ['./bin/python', 'setup.py', 'register'], ['./bin/python', 'setup.py', 'sdist', '--formats', 'zip', 'upload'])
else:
print ("\nSkipping the 'pypi' step since the the current version '%s' is already the stable version '%s'" % (current_version, args.stable_version))
print ("\nUploading version '%s' to PyPI" % args.stable_version)
# update version on github and add a tag
run_commands(None, ['./bin/python', 'setup.py', 'register'], ['./bin/python', 'setup.py', 'sdist', '--formats', 'zip', 'upload'])
if 'docs' in args.steps:
# Documentation can be uploaded, independent of the versions
print ("\nUploading documentation to PythonHosted.org")
run_commands(args.stable_version, ["./bin/python", "setup.py", "build_sphinx", "--source-dir", "doc", "--build-dir", "build/doc", "--all-files"], ["./bin/python", "setup.py", "upload_docs", "--upload-dir", "build/doc/html"])
run_commands(None, ["./bin/python", "setup.py", "build_sphinx", "--source-dir", "doc", "--build-dir", "build/doc", "--all-files"], ["./bin/python", "setup.py", "upload_docs", "--upload-dir", "build/doc/html"])
if 'latest' in args.steps:
# update GitHub version to latest version
print ("\nSetting latest version '%s'" % args.latest_version)
run_commands(args.latest_version, ['git', 'add', 'version.txt'], ['git', 'commit', '-m', '"Increased version to %s"' % args.latest_version], ['git', 'push'])
run_commands(args.latest_version, ['git', 'add', 'version.txt'], ['git', 'commit', '-m', 'Increased version to %s' % args.latest_version], ['git', 'push'])
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment