Skip to content
Snippets Groups Projects
Commit 0f5aa7cb authored by André Anjos's avatar André Anjos :speech_balloon:
Browse files

[doc] Redirect README to admin doc; Improve it

parent cb4d12ec
No related branches found
No related tags found
1 merge request!194Scheduler
Pipeline #
Expand all Hide whitespace changes
Inline Side-by-side
README.rst
+ 11
− 385
View file @ 0f5aa7cb
......@@ -25,8 +25,12 @@
Biometrics Evaluation and Testing Platform
============================================
This package contains the source code for the web components of the BEAT
platform.
The BEAT platform is written as a set of python packages. This package
(beat.web), in particular, constitutes the central deployment pillar of BEAT
platform instance. It uses as a base development library, a web framework
called Django_. If you are unfamiliar with this framework, but wishes to deploy
or develop the BEAT platform, it is recommended you familiarize yourself with
it.
Installation
......@@ -52,389 +56,15 @@ get you a fully operational test and development environment.
machinery.
.. tip::
If you'd like to **speed-up** the installation, it is strongly advised you
prepare a preset virtual environment (see the virtualenv_ package) with all
required dependencies, so that ``./bin/buildout`` does not download and
installs all of them every time you cleanup. This technique should allow you
to quickly clean-up and re-start your working environment which is useful
during development.
In order to fetch currently needed dependencies, run::
$ ./bin/buildout #to setup once
$ ./bin/pip freeze > requirements.txt
Examine the file ``requirements.txt`` and remove packages you are either
developing locally (e.g., all that are under ``src``) or that you think you
don't need. The command ``pip freeze`` reports all installed packages and not
only those which are needed by your project. If the Python prompt you used
for bootstrapping already had a good set of packages installed, you may see
them there.
Once you have a satisfying ``requirements.txt`` file, you may proceed to
recreate a virtualenv_ you'll use for your development. Just call::
$ virtualenv ~/work/beat-env #--system-site-packages
To create the virtual environment. This new environment does not contain
system packages by default. You may override that by specifying
``--system-site-packages`` as suggested above. Then, install the required
packages on your new virtual environment::
$ ~/work/beat-env/bin/pip install -r requirements.txt
After that step is done, your virtual environment is ready for deployment.
You may now start from scratch to develop ``beat.web`` taking as base the
Python interpreter on your virtualenv_::
$ cd beat.web
$ git clean -fdx #full clean-up
$ ~/work/beat-env/bin/python bootstrap-buildout.py
$ ./bin/buildout
You'll realize the buildout step now takes considerably less time and you may
repeat this last step as much as needed. ``pip`` is a very flexible tool and
you may use it to manage the virtualenv_ installing and removing packages as
needed.
Documentation
-------------
Our documentation project is divided in 3 parts. The user guide is the only one
which is automatically built as part of the ``buildout`` procedure. The API and
administrators guide need to be manually compiled if required.
To build the API documentation, just do::
$ ./bin/sphinx-apidoc --separate -d 2 --output=doc/api/api beat beat/web/*/migrations beat/web/*/tests
$ ./bin/sphinx-build doc/api html/api
To build the administrator guide, just do::
$ ./bin/sphinx-build doc/admin html/admin
The above commands will build the stated guides, in HTML format, and dump
results into your local directory ``html``. You may navigate then to that
directory and, with your preferred web browser, open the file ``index.html`` to
browse the available documentation.
The basic user guide which includes information for users of the platform, is
built automatically upon ``buildout``. If you wish to build it and place it
alongside the other guides, you may do it as well like this::
$ ./bin/sphinx-build doc/user html/user
Instantiating a BEAT web server
-------------------------------
For a simple (development) web server, the default settings on
``beat/web/settings/settings.py`` should work out of the box. These settings:
* Instantiate the web service on the local host under port 8000 (the address
will be ``http://127.0.0.1:8000``
* Use an SQLITE3 database named ``django.sql3`` located on the current
working directory
* Run with full debug output
* It sets the working BEAT prefix to ``./prefix``
* A single user, called ``user`` (password ``user``) will be setup into the
system
If you need to tweak these settings, just edit the file
``beat/web/settings/settings.py``. You may consult the `Django documentation`_
for detailed information on other settings.
Once the Django settings are in place, you can run a single command to fully
populate a development webserver::
$ ./bin/django install -v1
.. note::
Concerning databases installed by this command, we only explain the platform
how to **access** their data. It does not download the raw data for the
databases that you must procure yourself through the relevant web sites
(checkout the database pages on the Idiap instance of the BEAT platform for
details).
.. note::
If you need to specify your own path to the directories containing the
databases, you could just create a simple JSON file as follows::
{
"atnt/1": "/remote/databases/atnt",
"banca/2": "/remote/databases/banca"
}
Then just use the previous script with the option ``--database-root-file``::
$ ./bin/django install -v1 --database-root-file=MYFILE.json
By default, paths to the root of all databases are set to match the Idiap
Research Institute filesystem organisation.
.. note::
For every installed database, you'll need to generate their data indices,
which allows the platform to correctly parallelize algorithms. To do so, for
every combination of database and version you wish to support, run the
following command::
$ ./bin/beat -p prefix db index <name>/<version>
Replacing the strings ``<name>`` by the name of the database you wish to dump
the indices for, together with the version in ``<version>``. For example, to
dump the indices for the AT&T database, version 1, do the following::
$ ./bin/beat -p prefix db index atnt/1
Once the contributions and users are in place, you're ready to start the test
server::
$ ./bin/django runserver -v3
At this point, your platform can be accessed by typing the URL
``http://127.0.0.1:8000`` in a web browser on the machine the server is
running.
.. _localhost:
Localhost
---------
To effectively use your new server and test all aspects of it, you'll also need
a scheduler with at least one attached worker that can execute experiments. For
most development purposes, a simple 3-node system, with all components running
on the current (local) host is sufficient.
Here is a recipe to start a simple 3-node system in which the local worker uses
the system-wide installed Python interpreter to execute the algorithms.
First, make sure the program ``cpulimit`` is available on your system. The BEAT
platform uses this program to control slot usage on the scheduling/worker
level::
$ cpulimit -h
If that is not the case, then you need to install it. Either install a package
that is native to your system (e.g. on Debian or Ubuntu platforms) or compile
the checked-out version available at ``src/cpulimit``::
$ cd src/cpulimit;
$ make
$ ./src/cpulimit -h #to test it
$ cd ../../bin #go back to the root of beat.web and the into the `bin' dir
$ ln -s ../src/cpulimit/src/cpulimit
$ cd .. #go back to the root of beat.web
Now start the localhost system::
$ ./bin/localhost.py -v
...
You may inspect this programs help message for details on its usage and
options.
Once the localhost system is started you may open a browser window to your
`localhost, port 8000 <http://127.0.0.1:8000>`_, to get started with your
locally installed platform.
Localhost with ``DEBUG=False``
==============================
If you need to test the RESTful API, it is better to do it without Django
throwing you HTML error pages. For that, you'll need to start the Django
development server with slightly different settings::
$ ./bin/localhost.py -v --settings=beat.web.settings.nodebug
Unit Testing
------------
After installation, it is possible to run our suite of unit tests. To do so,
use::
$ ./bin/django test --settings=beat.web.settings.test -v 1
You may pass filtering criteria to just launch tests for a particular set of
``beat.web`` applications. For example, to run tests only concerning
``beat.web.toolchains``, run::
$ ./bin/django test --settings=beat.web.settings.test -v 1 beat.web.toolchains.tests
To measure coverage, you must set an environment variable for nose::
$ ./bin/coverage run --source='./beat/web' ./bin/django test --settings=beat.web.settings.test
$ ./bin/coverage report
Or, to generate an HTML report::
$ ./bin/coverate html
.. tip::
You may significatively speed-up your testing by re-using the same test
database from run to run. In order to do this, just specify the flag
``--keepdb`` when you run your tests::
$ ./bin/django test --settings=beat.web.settings.test -v 1 --keepdb
In this case, Django will create and keep a test database called
``test.sql3`` on your current directory. You may delete it when you're done.
.. _snapshot:
Local Development Server
------------------------
It is easy to quickly setup a local system for development, taking as base the
current state of a production system.
1. Before starting, make sure you have gone through, at least once, over the
localhost_ instructions above. It explains the very basic setup required for
a complete development environment.
2. Dump and back-up your current **production** BEAT database::
[production]$ ./bin/django backup
3. [Optional] If you have made important modifications between the contents
available at your production server and your currently checked-out source,
you'll need to run Django migrations on data imported from the production
server. If you need to do this, make sure you don't have unapplied commits
to your local **development** package and reset it to the production tag::
[development]$ git checkout <production-tag>
.. note::
You can figure you the production tag by looking at the footer of the
BEAT website. The corresponding tag name is found by prefixing a ``v``
before the version number. For example, the tag for version ``0.8.2`` of
the platform is ``v0.8.2``.
Also make sure to revert all dependent packages, so as to recreate the state
of the database schema as on the production site.
4. Remove the current local development database so that the restore operation
can start from scratch::
[development]$ rm -rf django.sql3 prefix
5. Copy the backup tarball from the production server and restore it locally::
[development]$ scp root@beatweb:backups/<backup-filename>.tar.bz2
[development]$ ./bin/django restore <backup-filename>.tar.bz2
At this point, you have recreated a copy of your production system locally,
on your SQLite3 database.
6. Reset queue configuration to allow for local running.
You may, optionally, reset the queue configuration of your installation so
that the environment you have is compatible with your development machine,
so that you can immediately run experiments locally. To do so, use the
``qsetup`` Django command::
[development]$ ./bin/django qsetup --reset
7. Apply migrations::
$ ./bin/django migrate
At this point, you should have a complete development setup with all elements
available on the production system installed locally. This system is fully
capable of running experiments locally using your machine. Start a full system
using ``localhost.py`` as explained on the localhost_ section above.
Testing Django Migrations
-------------------------
Django migrations, introduced in version 1.7, is a useful feature for
automatically migrating your database to new model schemas, if you get it
right. Here is a recipe to make sure your migrations will work on your
production system, allowing for quick and repetitive test/fix cycles.
The key idea is that we follow the setup for the snapshot_ and then,
locally backup our database and prefix so that we can quickly reproduce the
migration test loop.
1. Make sure you go through the snapshot_ instructions above (**up to
step 6 only**).
2. Make a copy of the SQLite3 database::
$ cp -a django.sql3 django.sql3.backup
This backup will allow you to quickly test the migrations w/o having to
checkout the production version anymore.
Also, create a temporary git repository of ``prefix``, so you can
cross-check changes and reset it in case of problems::
$ cd prefix
$ git init .
$ git add .
$ git commit -m "Initial commit"
$ cd ..
3. Go back to the HEAD or branch you were developping before::
$ git checkout HEAD
4. Here is how to test/fix your migrations:
a. Run "django migrate"::
$ ./bin/django migrate
b. Check your database by visually inspecting it on the django web admin or
by manually dumping it.
c. If a problem is detected, fix it and revert the state::
$ cp -af django.sql3.backup django.sql3
$ cd prefix && git reset --hard HEAD && git clean -fdx . & cd ..
.. note::
Tip: Write the above lines in a shell script so it is easy to repeat.
Go back to a. and restart.
Javascript Management with Node.js/Bower
----------------------------------------
We manage javascript external packages with the help of Bower_. If you'd like
to include more packages that will statically served with the Django web app,
please consider including them at the appropriate section of ``buildout.cfg``.
The included recipes will also download and install executables for
``uglifyjs``, ``grunt``, ``csslint`` and ``jshint``, which can be useful for JS
development.
Once the guide is built, continue reading on the "Installation" section for
further deployment instructions.
Issues
......@@ -444,13 +74,9 @@ If you find problems concerning this package, please post a message to our
`group mailing list`_. Currently open issues can be tracked at `our gitlab
page`_.
.. Place here references to all citations in lower case
.. _django documentation: https://docs.djangoproject.com
.. _pip: http://pypi.python.org/pypi/pip
.. _easy_install: http://pypi.python.org/pypi/setuptools
.. _zc.buildout: http://pypi.python.org/pypi/zc.buildout
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
.. _django: https://www.djangoproject.com/
.. _group mailing list: https://groups.google.com/d/forum/beat-devel
.. _our gitlab page: https://gitlab.idiap.ch/biometric/beat.web/issues
.. _bower: http://bower.io
.. _our gitlab page: https://gitlab.idiap.ch/beat/beat.web/issues
doc/admin/installation.rst
+ 339
− 174
View file @ 0f5aa7cb
This diff is collapsed.
doc/admin/platform_extension.rst
+ 1
− 1
View file @ 0f5aa7cb
......@@ -27,7 +27,7 @@ Extending the platform
======================
A running platform can be extended in several ways, depending to the specific
needs of this particular instance. We devise this section following a
needs of the particular instance. We devise this section following a
`Troubleshooting Guide` style. Counters installed on the BEAT platform software
allow for diagnosing and understanding of similar issues.
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment