Implemented a mechanism to run a dependency graph

This MR implements the calculation of dependence graphs for packages which are hosted in gitlab.

The command is available via bdt gitlab graph --help.

It recursively tries a package resolution via the conda-build API.

The user can choose the format of the output, which types of dependencies to track (among host, build, run and test) and what to put on the graph via a regular expression.

The user needs access to the gitlab API to use the command.

It replaces the old bob_dependency_graph command in bob.extension, that is now removed (see bob.extension!106 (merged)).

Partially addresses #43 (provides the adjancence matrix required for the evaluation of the DAG).

mentioned in issue bob.extension#62 (closed)

Will review this

assigned to @andre.anjos

You can also render dep. graph for beat components. For instance,

bdt graph beat.editor beat --prefix beat.

Generates

@tiago.pereira I thought this script will parse conda/meta.yaml files. Don't you want to use this for the release? What if dependencies are changed in this release? What about test requirements?

For the release I'm planning to use the beta channel for the conda search. Would be the most updated set of dependencies.

What conda search gives you is not the information needed to compile packages. conda search gives you only runtime requirements. How about host and test dependencies?

resolved all discussions

but that's what I need to parallelize the builds.

@tiago.pereira for example to build bob.bio.face, you need to build these packages first: https://gitlab.idiap.ch/bob/bob.bio.face/blob/7da637ace9146bc2ac5ec3cbc155863013655a59/conda/meta.yaml#L70 which don't show up in this script:

    - bob.db.arface
    - bob.db.atnt
    - bob.db.banca
    - bob.db.caspeal
    - bob.db.frgc
    - bob.db.gbu
    - bob.db.ijba
    - bob.db.ijbc
    - bob.db.lfw
    - bob.db.mobio
    - bob.db.msu_mfsd_mod
    - bob.db.multipie
    - bob.db.replay
    - bob.db.replaymobile
    - bob.db.scface
    - bob.db.xm2vts
    - bob.db.fargo
    - bob.bio.gmm
    - gridtk

Here are a few practical comments:

1. Our interface for the other scripts in this package waits for <group>/<package-name>, with group defaulting to bob if that is not mentioned. See an example here: https://gitlab.idiap.ch/bob/bob.devtools/blob/master/bob/devtools/scripts/getpath.py#L46
2. Channel dependency: the channel dependency will itself depend on the type of build. A stable build will use only stable channels. A build from the master tip will also deploy beta channels. The configuration for the channel resolution is implemented here: https://gitlab.idiap.ch/bob/bob.devtools/blob/master/bob/devtools/scripts/build.py#L188. In particular, that implementation explains how to hook the channel list into the conda-build API so you can use it in the next step. What I propose you do concretely, is to assume that the user wants to do a dependency graph on the master tip for the time being, which means you need to include the beta channels. We can then expand to run "stable" builds. To figure out if you need both private and public channels, use this trick: https://gitlab.idiap.ch/bob/bob.devtools/blob/master/bob/devtools/scripts/ci.py#L718
3. Now, in possession of the package group, name and list of channels, you should call the conda-build API to figure out the dependency list. You have an example of that implemented for a package here: https://gitlab.idiap.ch/bob/bob.devtools/blob/master/bob/devtools/build.py#L251. To call that function, you will need to do a git (shallow please) clone to a temporary directory. That will then resolve that package and you can extract a precise list of build, run and test time dependencies from there. Once you have that, you just have to recurse to resolve it for the ensemble (as you already do).

Instead of doing git clone, you can ask the gitlab API for the recipe file or maybe even the directory.

@tiago.pereira: any shots at this?

Soon. I think save the nightlies https://gitlab.idiap.ch/bob/bob.nightlies/issues/55 is more priority

snakemake as well

added 17 commits

• d5c5c7f7...23eb0125 - 16 commits from branch master
• a700c950 - Implemented a mechanism to run a dependency graph

Compare with previous version

added 3 commits

• 018e572d - [build] Avoid double-logging after importing conda modules
• 5e281f97 - [graph] Re-implement dependence graph generation as per my instructions to @tiago.pereira
• 7a3ba42d - [setup] Reset position of graph command

Compare with previous version

wow, nice :-)

ok. thank you

@tiago.pereira: Here are some samples.

You can obtain graphs such as this (./bin/bdt git graph -vv bob/bob.learn.linear):

Or more complex such as this (via setting the filtering regular expression or dependence types ./bin/bdt git graph -vv bob/bob.learn.linear -d run -w '.*'):

It also works for BEAT projects with the added advantage it tracks eventual dependencies to Bob packages as well:

The above graph considers all dependencies. Here is a version similar to yours, considering only "run" dependencies (-d run was used):

Note: I did not implement the DAG calculation, from the adjancence matrix. I'll let that to you, when you revisit the release building.