Skip to content

beat.examples
beat.examples

Home

Last edited by André Anjos
Page history

Creating a new BEAT Database

BEAT Databases are simply accessor plugins that allow the data belonging to a database to be read and fed into a BEAT Toolchain in the context of an Experiment. The accessors are called views in the BEAT jargon. Here is an UML diagram that explains the relationship of all components of a database successfuly integrated into the platform:

database-uml

Components

A BEAT database is composed of three main parts:

  1. A JSON file that declares the database protocols, its root folder location, sets, templates and outputs in terms of existing Dataformats;
  2. A RestructuredText document that thoroughly describes the database;
  3. A set of Python scripts that define the views (i.e., how to access the data), for each combination of protocol, set and output.

Procedure

To create a new BEAT database, you must be able to test the JSON descriptor (i.e., the arrangement of protocols, views, sets and outputs) together with your Python views inside, potentially, a real toolchain that uses the database. Doing so may be a complicated produce. This wiki page tries to simplify this process by providing instructions on how to build a new database from scratch.

Step 1: Create a Bob database satellite package and a simple toolchain

The very first step is to make sure you can use the database locally, via a Bob database satellite package. A simple toolchain can be used to test if the Bob database satellite package is working as expected. Instructions on how to create a Bob database satellite package are available here. You'll find numerous examples of database packages from that link.

Step 2: Install a development platform on your machine

Once your Python code is ready, the next thing to do is to install a developement platform on your machine. It will allow you to input the JSON description and insert the views at the right location for development. To do so, follow the instructions for development setup at our beat.web package. Follow the procedure to the end, including the (optional) installation of existing contributions.

Step 3: Add your database JSON and Views to beat.databases

Inside beat.web, you'll find a checkout of beat.examples in src/beat.examples. Go ahead and create the view and JSON descriptors matching the existing structure. Both JSON and the (python) View files should be uploaded to the BEAT server.

Note: If your Bob satellite package for the database is not available on PyPI, you must also add a checkout/mr.developer line on beat.web/buildout.cfg so that the system can properly download and install it from its repository.

Re-run ./bin/buildout at the beat.web level to make sure the new satellite package is installed and all is good for your first test.

To install the database into your development server, run ./bin/install_contributions -v advanced. If all is correct, the program should just report the insertion of the database into the system. Treat all errors until the database can be inserted correctly into the platform.

Step 4: Create a toolchain and experiment

With the database inserted into the system, launch beat.web this way:

$ ./bin/django runserver -v3

Then open your browser and point it to http://127.0.0.1:8000. Login as user user with password user then, create a new toolchain using your database template.

Once a toolchain that can use your database is in place, add the algorithms for such a toolchain and, finally, create an experiment mixing the database protocol you want to test and the algorithms. Do not push "Go", but just "Queue" the experiment for the time being.

Step 5: Run the experiment locally for interactive debugging

Once all is in place, you can quickly modify the views and algorithms so that the database works. The workflow is:

  1. Test run: ./bin/beat --prefix=web_dynamic_data experiments run <user>/<toolchain>/<version>/<experiment-label>
  2. Modify the view/algorithms
  3. Repeat from 1 until all works

Step 6: Commit to beat.databases

Once you are happy with all you did, you may commit the changes into the beat.examples package. You should, at least, include the database JSON, a RestructuredText description and the Python view, but it is also recommended you include a baseline toolchain, experiment and algorithms so that others can test your setup during development.

Note: If you have a database which is already in production at the BEAT platform, make sure that, when you modify your database, you create a new version.