Biometrics Evaluation and Testing Platform

This package contains the source code for the web components of the BEAT platform.

Installation

Here is a recipe to get you started:

$ python bootstrap-buildout.py
$ ./bin/buildout

These 2 commands should download and install all non-installed dependencies and get you a fully operational test and development environment.

$ ./bin/buildout #to setup once
$ ./bin/pip freeze > requirements.txt

$ virtualenv ~/work/beat-env #--system-site-packages

$ ~/work/beat-env/bin/pip install -r requirements.txt

$ cd beat.web
$ git clean -fdx #full clean-up
$ ~/work/beat-env/bin/python bootstrap-buildout.py
$ ./bin/buildout

Documentation

Our documentation project is divided in 3 parts. The user guide is the only one which is automatically built as part of the buildout procedure. The API and administrators guide need to be manually compiled if required.

To build the API documentation, just do:

$ ./bin/sphinx-apidoc --separate -d 2 --output=doc/api/api beat beat/web/*/migrations beat/web/*/tests
$ ./bin/sphinx-build doc/api html/api

To build the administrator guide, just do:

$ ./bin/sphinx-build doc/admin html/admin

The above commands will build the stated guides, in HTML format, and dump results into your local directory html. You may navigate then to that directory and, with your preferred web browser, open the file index.html to browse the available documentation.

The basic user guide which includes information for users of the platform, is built automatically upon buildout. If you wish to build it and place it alongside the other guides, you may do it as well like this:

$ ./bin/sphinx-build doc/user html/user

Instantiating a BEAT web server

For a simple (development) web server, the default settings on beat/web/settings/settings.py should work out of the box. These settings:

• Instantiate the web service on the local host under port 8000 (the address will be http://127.0.0.1:8000
• Use an SQLITE3 database named django.sql3 located on the current working directory
• Run with full debug output
• It sets the working BEAT prefix to ./prefix
• A single user, called user (password user) will be setup into the system

If you need to tweak these settings, just edit the file beat/web/settings/settings.py. You may consult the Django documentation for detailed information on other settings.

Once the Django settings are in place, you can run a single command to fully populate a development webserver:

$ ./bin/django install -v1

{
  "atnt/1": "/remote/databases/atnt",
  "banca/2": "/remote/databases/banca"
}

$ ./bin/django install -v1 --database-root-file=MYFILE.json

$ ./bin/beat -p prefix db index <name>/<version>

$ ./bin/beat -p prefix db index atnt/1

Once the contributions and users are in place, you're ready to start the test server:

$ ./bin/django runserver -v3

At this point, your platform can be accessed by typing the URL http://127.0.0.1:8000 in a web browser on the machine the server is running.

Localhost

To effectively use your new server and test all aspects of it, you'll also need a scheduler with at least one attached worker that can execute experiments. For most development purposes, a simple 3-node system, with all components running on the current (local) host is sufficient.

Here is a recipe to start a simple 3-node system in which the local worker uses the system-wide installed Python interpreter to execute the algorithms.

First, make sure the program cpulimit is available on your system. The BEAT platform uses this program to control slot usage on the scheduling/worker level:

$ cpulimit -h

If that is not the case, then you need to install it. Either install a package that is native to your system (e.g. on Debian or Ubuntu platforms) or compile the checked-out version available at src/cpulimit:

$ cd src/cpulimit;
$ make
$ ./src/cpulimit -h #to test it
$ cd ../../bin #go back to the root of beat.web and the into the `bin' dir
$ ln -s ../src/cpulimit/src/cpulimit
$ cd .. #go back to the root of beat.web

Now start the localhost system:

$ ./bin/localhost.py -v
...

You may inspect this programs help message for details on its usage and options.

Once the localhost system is started you may open a browser window to your localhost, port 8000, to get started with your locally installed platform.

Localhost with DEBUG=False

If you need to test the RESTful API, it is better to do it without Django throwing you HTML error pages. For that, you'll need to start the Django development server with slightly different settings:

$ ./bin/localhost.py -v --settings=beat.web.settings.nodebug

Unit Testing

After installation, it is possible to run our suite of unit tests. To do so, use:

$ ./bin/django test --settings=beat.web.settings.test -v 1

You may pass filtering criteria to just launch tests for a particular set of beat.web applications. For example, to run tests only concerning beat.web.toolchains, run:

$ ./bin/django test --settings=beat.web.settings.test -v 1 beat.web.toolchains.tests

To measure coverage, you must set an environment variable for nose:

$ ./bin/coverage run --source='./beat/web' ./bin/django test --settings=beat.web.settings.test
$ ./bin/coverage report

Or, to generate an HTML report:

$ ./bin/coverate html

$ ./bin/django test --settings=beat.web.settings.test -v 1 --keepdb

Local Development Server

It is easy to quickly setup a local system for development, taking as base the current state of a production system.

1. Before starting, make sure you have gone through, at least once, over the localhost instructions above. It explains the very basic setup required for a complete development environment.

2. Dump and back-up your current production BEAT database:

   [production]$ ./bin/django backup

3. [Optional] If you have made important modifications between the contents available at your production server and your currently checked-out source, you'll need to run Django migrations on data imported from the production server. If you need to do this, make sure you don't have unapplied commits to your local development package and reset it to the production tag:

   [development]$ git checkout <production-tag>

   Note

   You can figure you the production tag by looking at the footer of the BEAT website. The corresponding tag name is found by prefixing a v before the version number. For example, the tag for version 0.8.2 of the platform is v0.8.2.

   Also make sure to revert all dependent packages, so as to recreate the state of the database schema as on the production site.

4. Remove the current local development database so that the restore operation can start from scratch:

   [development]$ rm -rf django.sql3 prefix

5. Copy the backup tarball from the production server and restore it locally:

   [development]$ scp root@beatweb:backups/<backup-filename>.tar.bz2
   [development]$ ./bin/django restore <backup-filename>.tar.bz2

   At this point, you have recreated a copy of your production system locally, on your SQLite3 database.

6. Reset queue configuration to allow for local running.

   You may, optionally, reset the queue configuration of your installation so that the environment you have is compatible with your development machine, so that you can immediately run experiments locally. To do so, use the qsetup Django command:

   [development]$ ./bin/django qsetup --reset

7. Apply migrations:

   $ ./bin/django migrate

At this point, you should have a complete development setup with all elements available on the production system installed locally. This system is fully capable of running experiments locally using your machine. Start a full system using localhost.py as explained on the localhost section above.

Testing Django Migrations

Django migrations, introduced in version 1.7, is a useful feature for automatically migrating your database to new model schemas, if you get it right. Here is a recipe to make sure your migrations will work on your production system, allowing for quick and repetitive test/fix cycles.

The key idea is that we follow the setup for the snapshot and then, locally backup our database and prefix so that we can quickly reproduce the migration test loop.

1. Make sure you go through the snapshot instructions above (up to step 6 only).

2. Make a copy of the SQLite3 database:

   $ cp -a django.sql3 django.sql3.backup

   This backup will allow you to quickly test the migrations w/o having to checkout the production version anymore.

   Also, create a temporary git repository of prefix, so you can cross-check changes and reset it in case of problems:

   $ cd prefix
   $ git init .
   $ git add .
   $ git commit -m "Initial commit"
   $ cd ..

3. Go back to the HEAD or branch you were developping before:

   $ git checkout HEAD

4. Here is how to test/fix your migrations:

   1. Run "django migrate":

      $ ./bin/django migrate

   2. Check your database by visually inspecting it on the django web admin or by manually dumping it.

   3. If a problem is detected, fix it and revert the state:

      $ cp -af django.sql3.backup django.sql3
      $ cd prefix && git reset --hard HEAD && git clean -fdx . & cd ..

      Note

      Tip: Write the above lines in a shell script so it is easy to repeat.

      Go back to a. and restart.

Javascript Management with Node.js/Bower

We manage javascript external packages with the help of Bower. If you'd like to include more packages that will statically served with the Django web app, please consider including them at the appropriate section of buildout.cfg.

The included recipes will also download and install executables for uglifyjs, grunt, csslint and jshint, which can be useful for JS development.

Issues

If you find problems concerning this package, please post a message to our group mailing list. Currently open issues can be tracked at our gitlab page.