Tight "soft-loops" controlling algorithmic training

In the context of the ALLIES project, a so-far-avoided requirement has popped up again the meeting: block loops.

I created this bug to differentiate from #29, because the requirements here are a bit different: in the use-case to be considered, platform users need to be able to train a model depending on the output of another block. The main research question of the project needs to answer is: is it possible to implement life-long learning systems (LLLS) that can be taught by a human and improve over time? The outcome of the project must be reproducible (therefore our involvement).

In order to simulate this, partners provided the following diagram:

On the left, you'll see conventional ways to "adapt" statistical models from new input data. Given the model and more data, the model can be adapted to new incoming information (a.k.a. "source" in these diagrams).

In the context of ALLIES, partners want to have reproducible toolchains that allow third-parties to create the training system on the right of this image. In this context, the yellow boxes under the "training" (cyan) boxes simulate a human-user interaction with the system. E.g.: it could be a human telling a machine translation system the translation of a sentence (or paragraph) needs to be corrected, alongside with the correction, or it could be a human telling a speaker diarization system that a certain segment belongs to a different speaker. Because humans have limited ability to provide corrections, the consortium wants to control the implementation and limits of the "yellow" boxes, while third-parties implement the "trainers" that go into the cyan boxes.

It is possible to "unroll" these loops for practical scenarios, but in order to test limits of LLLS, one can envisage hundreds if not thousand iterations which is not very practical to code in BEAT as of today.

One of the key questions while considering #29 was that of the cache: a system that loops will issue multiply different cache outputs which, using the current architecture, would have to be kept. In this new use-case, such a caching requirement is not present. Partners only want to ensure that the code in "yellow" boxes can be properly tagged by the consortium and that they can conduct searches and build leaderboards on experiments using those (and only those). This prevents implementors of the "cyan" training boxes to misinterpret the role and data from the simulated human (yellow boxes). As a consequence, no caching is required within this loop.