Creating a new BEAT Database
BEAT Databases are simply accessor plugins that allow the data belonging to a database to be read and fed into a BEAT Toolchain in the context of an Experiment. The accessors are called views in the BEAT jargon. Here is an UML diagram that explains the relationship of all components of a database successfuly integrated into the platform:
Components
A BEAT database is composed of three main parts:
- A JSON file that declares the database protocols, its root folder location, sets, templates and outputs in terms of existing Dataformats;
- A RestructuredText document that thoroughly describes the database;
- A set of Python scripts that define the views (i.e., how to access the data), for each combination of protocol, set and output.
Procedure
To create a new BEAT database, you must be able to test the JSON descriptor (i.e., the arrangement of protocols, views, sets and outputs) together with your Python views inside, potentially, a real toolchain that uses the database. Doing so may be a complicated produce. This wiki page tries to simplify this process by providing instructions on how to build a new database from scratch.
Step 1: Create a Bob database satellite package and a simple toolchain
The very first step is to make sure you can use the database locally, via a Bob database satellite package. A simple toolchain can be used to test if the Bob database satellite package is working as expected. Instructions on how to create a Bob database satellite package are available here. You'll find numerous examples of database packages from that link.
Step 2: Install a development platform on your machine
Once your Python code is ready, the next thing to do is to install a developement platform on your machine. It will allow you to input the JSON description and insert the views at the right location for development. To do so, follow the instructions for development setup at our beat.web package. Follow the procedure to the end, including the (optional) installation of existing contributions.
Step 3: Add your database JSON and Views to beat.databases
Inside beat.web
, you'll find a checkout of beat.examples
in src/beat.examples
. Go ahead and create the view and JSON descriptors matching the existing structure. Both JSON and the (python) View files should be uploaded to the BEAT server.
Note: If your Bob satellite package for the database is not available on PyPI, you must also add a checkout/mr.developer line on
beat.web/buildout.cfg
so that the system can properly download and install it from its repository.
Re-run ./bin/buildout
at the beat.web
level to make sure the new satellite package is installed and all is good for your first test.
To install the database into your development server, run ./bin/install_contributions -v advanced
. If all is correct, the program should just report the insertion of the database into the system. Treat all errors until the database can be inserted correctly into the platform.
Step 4: Create a toolchain and experiment
With the database inserted into the system, launch beat.web
this way:
$ ./bin/django runserver -v3
Then open your browser and point it to http://127.0.0.1:8000
. Login as user user
with password user
then, create a new toolchain using your database template.
Once a toolchain that can use your database is in place, add the algorithms for such a toolchain and, finally, create an experiment mixing the database protocol you want to test and the algorithms. Do not push "Go", but just "Queue" the experiment for the time being.
Step 5: Run the experiment locally for interactive debugging
Once all is in place, you can quickly modify the views and algorithms so that the database works. The workflow is:
- Test run:
./bin/beat --prefix=web_dynamic_data experiments run <user>/<toolchain>/<version>/<experiment-label>
- Modify the view/algorithms
- Repeat from 1 until all works
beat.databases
Step 6: Commit to Once you are happy with all you did, you may commit the changes into the beat.examples
package. You should, at least, include the database JSON, a RestructuredText description and the Python view, but it is also recommended you include a baseline toolchain, experiment and algorithms so that others can test your setup during development.
Note: If you have a database which is already in production at the BEAT platform, make sure that, when you modify your database, you create a new version.