- Up-to-date Chrome browser for Chrome headless testing support (doesn't work on Chromium yet)
- Up-to-date Firefox browser for Firefox headless testing support
#### Editor Support
- Make sure to pick up JS & JSX syntax highlighting for your editor!
- The provided `.tern-project` file is configuration for Tern.js, a JS code analysis tool (autocompletion, etc.).
- Linting via ESLint should be configured to use the local installation (from `node_modules/`) - *not* a global installation. This lets ESLint properly find all the plugins on a per-project basis.
- Linting CSS via Stylelint should be configured similarly to ESLint. Since this linter is relatively new, your linting engine/editor probably won't pick it up automatically.
#### Development
- You'll probably want the [React devtools](https://github.com/facebook/react-devtools) and [Redux devtools](https://github.com/zalmoxisus/redux-devtools-extension) browser plugins to sanely debug and inspect React/Redux code.
- You might need to enable/configure sourcemaps in your browser manually.
- The server file is `server.py`. To start it, you need the proper dependendencies in the python environment you use to run the server (see the `server.py` header for more info).
- You'll probably want Webpack's dev server (`npm start`) running in a separate terminal window/tab, as Webpack will automatically rebuild/update your project when files change (served at `localhost:9101`. However, changes to the webpack config or other configuration files may not be watched. In general, anything under `src/` should be hot-reloaded, while anything else may not be. If there's issues with the app when running the dev server that you don't expect or don't make sense (especially if the issue came up after a hot reload), try refreshing the page (`ctrl-r`) - sometimes the hot reloader doesn't properly handle complex changes (such as state management or async functionality).
- When changing/updating dependencies, you'll want to install/update flow types for them. You'll need a globally-installed `flow-typed` (`npm install -g flow-typed`) to run `flow-typed install` in the root of the project.
- If you're quickly iterating test code, you can run the testing daemon in a tmux/byobu pane with the `test-start` npm script. This lets you quickly trigger tests again with `karma run` (via a global `karma-cli` installation through npm) since it won't have to start up the daemon each time you want to run the tests. It will also rerun tests when the files change (this is configurable). The daemon is a bit buggy, so if you are getting unexpected errors, try to restart it.
## NPM Scripts Explanations
| Script | Target | Purpose |
| :----: | :----: | :------ |
| `start` | `src/main.jsx` | Your actual project |
| `prebuild` | `dist/` | Clears `dist/` of old files |
| `build` | `src/main.js` | Builds the whole app for production into `dist/` |
| `test` | `karma.conf.js` | Runs tests in `test/` via Karma |
| `test-start` | `karma.conf.js` | Runs tests in `test/` via Karma and keeps the test daemon running in the foreground. |
| `lint` | | Checks your JS, JSX, and CSS files for any errors |
| `lint:fix` | | Fixes any easier linting issues in your JS & JSX files |
| `flow` | `.flowconfig` | Checks your project's flow type annotations (static analysis) |
| `prepush` | `karma.conf.js` | Automatically runs before pushing (Git hook integration provided by `husky`) |
| redux | State Management library | Gives us a clear way to manage state across the entire app |
| redux-thunk | Redux action support for async operations | Lets Redux use async code (e.g. fetching from our server) |
| reselect | Memoized Redux selectors | Lets us calculate intermediate values from the Redux store once, even if the same calculation is used several places |
### Development Dependencies
| Package | Purpose | Simple Explanation |
| :------ | :------ | :----------------- |
| babel-core | Compiler | Core of Babel, an ecosystem for compiling JS code (e.g. transforming new features to older features to support old browsers) |
| babel-eslint | Compiler support for linter | Lets ESLint parse advanced features that aren't built into ESLint's default parser |
| babel-loader | Compiler support for Webpack | Lets Webpack pass JS files through Babel when building |
| babel-plugin-dynamic-import-webpack | Compiler support for dynamic `import`s | Lets Webpack process `import`s called anywhere in code |
| babel-plugin-istanbul | Babel support for Istanbul | Lets Babel track which code is being ran as it's being compiled |
| babel-plugin-syntax-dynamic-import | Babel support for the dynamic import syntax | Lets Babel parse `import` statements in arbitrary places |
| babel-plugin-syntax-object-rest-spread | Babel support for the object spread syntax | Lets Babel parse using spread syntax (`...`) for objects as well as arrays |
| babel-plugin-transform-object-rest-spread | Babel support for transforming object spread | Lets Babel transform the object spread syntax into something compatible with older JS engines |
| babel-preset-env | Babel support for dynamic targeting via browserslist syntax | Lets Babel dynamically change what code it changes based on your configuration |
| babel-preset-react | Babel support for JSX, Flow types, and other react things | Lets Babel understand & process idiomatic React code |
| chai | Assertion library | Provides tools for `assert`ing things in tests |
| chai-enzyme | Enzyme integration into chai | Lets you use enzyme assertion/testing functionality in chai statements instead of complex enzyme queries |
| cross-env | Cross-environment variable support | Transforms environment variable definitions written in Linux format to the current platform |
| enzyme-adapter-react-16 | Enzyme support for React 16 | |
| enzyme | React test utilities | Adds helpers for testing React code, especially Components |
| eslint | JS Linter | Statically analyzes JS code according to your configuration |
| eslint-plugin-compat | ESLint support for browserslist | Lets ESLint check if the JS APIs you use are supported by the browsers set in your configuration |
| eslint-plugin-flowtype | ESLint support for flow | Lets ESLint check flow type annotations (across files too!) |
| eslint-plugin-import | ESLint support for dynamic `import`s | Lets ESLint parse `import`s anywhere in JS code |
| eslint-plugin-react | ESLint support for React | Lets ESLint parse & lint React code & JSX |
| flow-bin | Flow type annotation linting/checking support | Adds support for Flow type annotations in your code (opt-in static typing for JS!) |
| flow-typed | Tool to use type defs from `flow-typed` | Lets you download & use high-quality community-written flow type definitions for libraries |
| html-webpack-harddisk-plugin | Lets HtmlWebpackPlugin always write to disk | |
| html-webpack-plugin | Generates `index.html` files with all chunks included | |
| husky | Calls npm scripts on git hooks | |
| karma | Test runner | Runs tests on a certain environment according to configuration and reports on results |
| karma-chrome-launcher | Karma support for Chrome browser | Lets Karma run tests in the Chrome browser, if installed |
| karma-coverage | Karma support for coverage instrumentation | Lets karma read output from Istanbul |
| karma-firefox-launcher | Karma support for Firefox browser | Lets Karma run tests in the Firefox browser, if installed |
| karma-mocha | Karma support for mocha | Lets Karma understand tests written in the Mocha testing framework |
| karma-mocha-reporter | Karma support for mocha reporters | Lets Karma use mocha reporters |
| karma-sourcemap-loader | Karma support for sourcemaps | Errors during testing are shown on the actual line they came from, not the line in the compiled files |
| karma-webpack | Karma support for webpack | Lets Karma run test files through webpack before running tests |
| mocha | Unit Test Framework | Lets us run unit tests |
| postcss | CSS compiler | Like Babel, but for CSS |
| postcss-cli | CLI for PostCSS | Lets us run PostCSS in the commandline not just in JS |
| postcss-cssnext | PostCSS support for latest CSS features | Lets us use new CSS features that may not be available in all browsers |
| postcss-loader | Webpack support for PostCSS | Lets Webpack run CSS through PostCSS |
| postcss-smart-import | Smarter importing logic for CSS | |
| rimraf | Small helper for emptying a directory | |
| sinon | Spies, stubs, mocks for JS testing | Lets you test callbacks, stub libraries with "fake" functions, mock server data |
| stylelint | CSS Linter | Like ESLint, but checks our CSS code instead of JS |
| stylelint-config-standard | Stylelint standard config | Gives sane default for Stylelint |
| svg-inline-loader | Webpack support for SVG files | Makes Webpack resolve `import <file>.svg` statements with the actually string from the SVG file |
| uglifyjs-webpack-plugin | Webpack plugin for minification | Makes webpack remove unused code and shrink it |
| webpack | Build system | Processes & bundles your source files and dependencies together |
| webpack-dev-server | Serves a webpack project on a port & rebuilds when necessary | |
| webpack-visualizer-plugin | Visualizer for webpack chunks | Provides a webpage for inspecting the contents of the generated webpack chunks |
| worker-loader | Webpack support for shared/service workers | Lets you use JS files that will be executed in a worker |
This section is a compilation of notes and advice about how to develop the ``beat.editor`` project, aimed at those less familiar with web development but familiar with Python.
Preparing the package for development
=====================================
The requirements for installing and preparing the ``beat.editor`` package for development are as following:
#. Linux or MacOS
#. Working internet connection
#. Useable Conda setup (see `Bob's documentation on it <https://www.idiap.ch/software/bob/docs/bob/bob/master/install.html>`_)
#. Up-to-date Firefox and Chrome browsers (both need headless support, a relatively recent feature)
#. Docker installed and available to the current user
#. A relatively recent version of `NodeJS & NPM <http://nodejs.org/>`_.
#. Activate the created environment and use the ``buildout`` command from ``bob.buildout`` in the new environment to generate the executables for ``beat.editor``: ::
$ conda activate beatedit
$ buildout
#. Note for development mode: Add the line ``dependent-scripts = true`` then use the ``buildout`` command to ensure to generate all the executables in the ``bin/``
folder even ``beat`` for ``beat.editor``: ::
$ buildout
#. In ``bin/`` you should several new executables for the project. The Python setup part is done.
#. Go into the root folder for the Javascript and install the dependencies via NPM: ::
$ cd conda/js
$ npm install
#. Test the JS to make sure everything is working properly: ::
$ npm test
#. Assuming the tests pass, you can start the development webpack server: ::
$ npm start
#. In a separate terminal tab/window, go back to the root of the ``beat.editor`` project and run the web server in development mode: ::
$ ./bin/beat editor serve --dev
#. You should now be able to go ``localhost:9101`` in your browser to see the ``beat.editor`` web app, now served by the webpack dev server.
Important Concepts
==================
Before developing for ``beat.editor``, you'll need to familiarize yourself with at least the following concepts/tools:
* Flask & RESTful APIs
* JSON
* JSON-schema, v4
* Node.js
* NPM
* HTML
* CSS
* The DOM & browser APIs
* Javascript
* ES6
* JSX
* React
* Redux
* Bootstrap
* D3 & SVG in general
* Flow typings
* Webpack
* Babel
* Karma
* Mocha
Configuring Your Editor
-----------------------
Your editor is probably adequately configured for editing Python code; it won't be properly configured for the JS part of ``beat.editor`` unless you do modern web development.
* Make sure to pick up JS & JSX syntax highlighting for your editor! Make sure your highlighting supports "ES6" (a landmark JS verion released a couple years ago) and "Flow" (the static typing language extension we are using).
* The configuration files for various linters/static analysis tools are found as dotfiles in the root folder for the JS (next to ``package.json``).
* The provided ``.tern-project`` file is configuration for Tern.js, a JS code analysis tool (autocompletion, etc.). If you're using a popular editor, it'll probably have a plugin for Tern.js.
* There's a few diferent linters available for the different code types (JS, HTML, CSS, JSX). I would suggest installing a linting engine that picks up the available linters for you, like "Syntastic" or "ALE" for Vim. If you aren't using a linting engine, here's some additional notes to get linting working:
- Linting via ESLint should be configured to use the local installation (from ``node_modules/``) - *not* a global installation. This lets ESLint properly find all the plugins on a per-project basis.
- Linting CSS via Stylelint should be configured similarly to ESLint. Since this linter is relatively new, your linting engine/editor might not pick it up automatically.
* Linting using Flow needs more configuration regardless of how you set up linting. You need to install the ``flow-typed`` package from NPM globally and pull the types for dependencies: ::
$ npm i -g flow-typed
$ cd conda/js
$ flow-typed update
Development Notes
-----------------
- You'll probably want the `React devtools <https://github.com/facebook/react-devtools>`_ and `Redux devtools <https://github.com/zalmoxisus/redux-devtools-extension>`_ browser plugins to sanely debug and inspect React/Redux code.
- You might need to enable/configure sourcemaps in your browser manually.
- The Python server can be run in dev mode (doesn't try to launch a browser tab) via the ``--dev`` flag.
- You'll probably want Webpack's dev server (``npm start``) running in a separate terminal window/tab, as Webpack will automatically rebuild/update your project when files change (served at ``localhost:9101``. However, changes to the webpack config or other configuration files may not be watched. In general, anything under ``conda/js/src/`` should be hot-reloaded, while anything else may not be. If there's issues with the app when running the dev server that you don't expect or don't make sense (especially if the issue came up after a hot reload), try refreshing the page - sometimes the hot reloader doesn't properly handle complex changes (such as state management or async functionality).
- When changing/updating dependencies, you'll want to install/update flow types for them. Run ``flow-typed update`` again whenever you add JS dependencies.
- If you're quickly iterating test code, you can run the testing daemon in a terminal tab/pane with the ``test-start`` npm script. This lets you quickly trigger tests again with ``karma run`` (via a global ``karma-cli`` installation through npm) since it won't have to start up the daemon each time you want to run the tests. It will also rerun tests when the files change (this is configurable). The daemon is a bit buggy, so if you are getting unexpected errors, try restarting it.
JS Project Structure
--------------------
All JS is stored under ``conda/js``. Unless otherwise noted, all future paths given in this section are relative to that one.
* All dependencies are stored in ``node_modules/``.
* All source is stored in ``src/``.
* All ``*.spec.*`` files are test files. They should be next to the file/component they are testing, which has the same name minus the ``.spec`` infix.
* All this source code is pulled together by ``main.jsx``, which is processed by Webpack via the ``webpack.config.js`` config file.
* ``test/`` holds test data and configures and runs the tests.
* There are 3 different folders in ``src/``, ``components/``, ``helpers``, and ``store/``. These three different folders are discussed further below.
components/
***********
``components/`` holds the UX components, written in a mix of HTML & JSX & CSS, and their associated processing code. Non-generic components (components related to a specific editor) are stored in their respective sub-folders, broken up by BEAT entity type:
* ``algorithm/`` holds the Algorithm Editor code.
* ``database/`` holds the Database Editor code.
* ``dataformat/`` holds the Dataformat Editor code.
* ``experiment/`` holds the Experiment Editor code, which also uses code from the toolchain editor.
* ``library/``: holds the Library Editor code.
* ``plotter/``: holds the Plotter Editor code, which is very similar to the Algorithm Editor.
* ``plotterparameter/``: holds the Plotterparameter code.
* ``toolchain/``: holds the Toolchain Editor code, which is by far the most complex module. The "Graphical Editor" files and the "ToolchainConnection" and "ToolchainBlock" hold most of the code concerned with actual SVG drawing/interaction, while the other files hold the processing code and methods to connect the Graphical Editor to the data.
Generic components, such as the components for the list views, navigation (using `react-router v4 <https://reacttraining.com/react-router/web/guides/philosophy>`_, or common sub-components are just stored in ``components/``.
helpers/
********
``helpers/`` holds code used in many different places or generic utility code. The ``helpers/schema/`` subfolder holds the code & schemas for validating BEAT objects via `JSON schema validation <http://json-schema.org/>`_ (BEAT uses Draft-04). There's a schema for each entity type.
To talk to the ``beat.editor`` REST server, one would use the contents in the ``api`` helper file. For BEAT-wide helpers, see the ``beat`` helper.
store/
******
``store/`` holds all the code for the Redux store, including the reducers, actions, and selectors. We use `reselect <https://github.com/reduxjs/reselect>`_ to write memoized composable selectors, in ``selectors.js``.
REST API
--------
How does the webapp operate on the local BEAT prefix? Through the small Python REST API server provided in ``beat/editor/`` in this project. This is a bare-bones REST API server using `Flask <http://flask.pocoo.org/>`_ that exposes the following API on ``localhost:5000``:
* ``/`` or ``/index.html``: Serves the production/distributable version of the webapp.
* ``<plural BEAT entity name>/``: For each type of BEAT object, there is an endpoint for it. This endpoint is the pluralized version of the type - to operate on databases, use ``/databases``, to operate on libraries, use ``/libraries``, etc. Each endpoint accepts the following HTTP verbs:
- ``GET``: GET requests fetch all the objects of the given type from the prefix.
- ``POST``: POST requests create an object given the ``obj`` field, the object to create, and the ``copiedObjName`` field, which is an optional field to specify the object being copied.
- ``PUT``: PUT requests update objects, overwriting the objects in the prefix with the given objects (matched by name).
- ``DELETE``: DELETE requests delete objects from the prefix.
* ``settings/``: Only accepts GET requests - fetches settings (not currently being used).
* ``environments/``: Only accepts GET requests - fetches the docker environments (can be slow because ``beat.core`` needs to query every docker container).
* ``layout/``: Only accepts POST requests - given the toolchain as the request body, generates a layout for the toolchain using Graphviz's ``dot`` layout algorithm and returns it.
.. automodule:: beat.editor
E2E Testing
-----------
There are selenium tests found in ``conda/js/test/``. These tests are set up to be ran in Firefox in headless mode with the REST server running locally. To run these tests:
* A relatively recent version of Firefox with headless support
* You need the contents of the tutorial's prefix in your local BEAT prefix (find it at ``https://gitlab.idiap.ch/beat/beat.tutorial.prefix``)
* A recent version of `the Geckodriver executable <https://github.com/mozilla/geckodriver/releases/>`_ available in your path for Selenium to use
* The ``beat.editor`` REST server running locally
Just do ``node conda/js/test/<selenium test>`` to run the test. Please see inside the tests for additional notes.
The tests should always be cleaning up test artifacts in your prefix after the test finishes. If tests do not finish successfully, some of these artifacts may still be present in your prefix and will cause future runs of that test to fail. So, if a test doesn't finish successfully, you will have to delete the test artifacts manually. To make it easier, all BEAT objects created by these tests have the username "selenium" so you know what to delete.
Developing E2E Tests
********************
The webdriver & all its functionality is `well documented <http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/>`_.
It's recommended to debug tests by not using headless mode and inserting plenty of long pauses via ``driver.sleep()``. You'll need to know the modern ``async``/``await`` pattern as well as be comfortable with CSS selector syntax. See the ``selenium_tutorial_test.js`` test file for working examples of these concepts and how to use selenium's API.
This package provides a web server to manage and edit BEAT objects locally. The users can make new objects or new versions of existing ones using the graphical interface provided by this server and the editor produces the correct JSON_ format for the corresponding objects. In case of new objects it also provides a Python template as well. This is the package used as the graphical interface in
This package provides a web server to manage and edit BEAT objects locally. The users can make new objects or new versions of existing ones using the graphical interface provided by this server and the editor produces the correct JSON_ format for the corresponding objects. In case of new objects it also provides a Python template as well. This is the package used as the graphical interface in
`A Hands On Tutorial <https://www.idiap.ch/software/beat/docs/beat/docs/master/beat/user.html>`_.
In the following sections different parts of the graphical interface will be explained in details. The developers interested in developing this package can find the information in :ref:`beat-editor-development`.
In the following sections different parts of the graphical interface will be explained in details. The developers interested in developing this package can find the information in :ref:`beat-editor-contribute`.
