Skip to content

bob.devtools
bob.devtools
Tags give the ability to mark specific points in history as being important
  • v3.1.6
    2736a281 · Increased stable version to 3.1.6 ·
    Release v3.1.6
    • !168 Give public packages more priority over private packages
    • !169 Downgrade twine: The current version (3.2.0) installation misses two dependencies when using conda to install it. As requested by @amohammadi, downgrade to a version that does not have that issue.
  • v3.1.5
    53bfbe37 · Increased stable version to 3.1.5 ·
    Release v3.1.5
    • !163 [pre-commit] Add pre-commit to bob.devtools and its template
    • !164 Implement hot-fix to repo indexing: Adds a conda package that when uploaded to our channel, patches the repodata. Fixes #53 and #56
    • !165 Prevent noarch packages from being deployed from mac builds: Closes #57
    • !166 [repodata-patches] remove kaldi packages: This will allow us to remove epoch numbers from Kaldi versions which are troublesome: https://github.com/conda/conda-build/issues/2926
    • !167 Update Kaldi version
  • v3.1.4
    fccdd04a · Increased stable version to 3.1.4 ·
    Release v3.1.4
    • !146 Fix version check regular expressions in relase script: One was not properly escaping the . and the other one was neither escaping the . nor handling more than one digit numbers. This results in the release process crashing as soon as a package version number item hits 10 when using the automatic increment.
    • !147 Bump nightly requirements for Bob and BEAT: Multiple package version upgrades.
    • !149 Update base version of miniconda: Use version 4.8.2 of the miniconda installer.
    • !150 Updates python-gitlab to 2.1.2
    • !151 Remove custom patch to Sphinx doc generation to avoid private members: Since sphinx 2.x, no need to avoid private functions explicitly anymore. Everything is respected adequately and we do not need this patch anymore.
    • !153 Fix badge URL: This MR clarifies what needs to be done to tackle #51.
    • !152 Add coverage reporting for single-package builds: * Changes the file single-package.yml to allow gitlab to interpret coverage reports in merge requests. * Provides a template for the-right-way to do coverage report * Hooks the coverage report visualisation to the test coverage badge in the template README
    • !154 Add script to automatically update badges from gitlab projects: A new command is available under the gitlab subcommand. It can create/update README badges for a project automatically. To use it, just issue bdt gitlab badges <group>/<project> -vv It partially solves #51 by providing a tool to automatically update repos in a consistent manner.
    • !156 Add more dask packages and xarray to conda_build_config.yaml
    • !155 Add dask to conda_build_config.yaml
    • !157 Bump pytorch/torchvision bob/beat-devel: New pytorch (1.3.1), torchvision (0.4.2) and cudatoolkit (10.0).
    • !158 [build,data] Cache eventual pytorch checkpoints used in unit testing: This MR will make our CI cache models downloaded during pytorch tests.
    • !148 Python 3.8 support: This MR introduces preliminary support for Python 3.8 for the basic development tool.
    • !159 Let conda-build handle python variants: So far, we used to explicitly set the target python version to build for packages listed in python.txt at bob/bob.conda>. With this MR, this is deprecated - adding support for newer versions of Python gets less messy this way.
    • !160 Fix base_build() with a proper support for multi-package-building through conda-build: This MR fixes build.base_build() to properly handle building multiple versions of the same package according to what is defined in conda_build_config.yaml.
    • !162 Resolve "undefined variable in build.py": Closes #54
    • !161 Update conda build config to match the defaults channel: This MR probably requires an update to our mac runners as well to install 10.10 sdk.
  • v3.1.3
    1e4bab12 · Increased stable version to 3.1.3 ·
    Release v3.1.3
    • !144 Fixes pypi stage to skip deployment if current package is private: Closes #50
    • !145 Improve the release scripts: Fixes #49 #29
  • v3.1.2
    9ddcee24 · Increased stable version to 3.1.2 ·
    Release v3.1.2
    • !107 [docs] acumulating information for bob package development from other packages: In this merge request all the information about local development of bob packages will be added and updated. These information have been in the documentation of bob.buildout and bob.extension before.
    • !143 [conda_build_config] Sync with removal of dependencies in beat-devel: Items removed: - flask - flask-cors - flask-restful - nodejs see bob.conda!438
    • !127 Reset license file location and remove hack (closes #44): This MR: 1. Removes the hack affecting the builds 2. Corrects the location for the own conda recipe 3. Corrects the location to be used for bdt new runs
  • v3.1.1
    3113d515 · Increased stable version to 3.1.1 ·
    Release v3.1.1
    • !142 Fix bug - "ci" does not exist: This MR fixes a small error on calling get_channels the create command.
  • v3.1.0
    15f9ee1b · Increased stable version to 3.1.0 ·
    Release v3.1.0
    • This update brings the use of "strict" channel priority in conda for resolving package dependencies.
    • !107 [docs] acumulating information for bob package development from other packages: In this merge request all the information about local development of bob packages will be added and updated. These information have been in the documentation of bob.buildout and bob.extension before.
    • !129 Implements a generic mechanism to avoid slow tests if $NOSE_EVAL_ATTR is set.: Allows the environment variable $NOSE_EVAL_ATTR to indicate which tests to skip during conda build (c.f. bob.nightlies#55)
    • !130 Set default channel-priority to "strict" due to issues with beta...: Set default channel-priority to "strict" due to issues with beta builds using mixed stable/beta sources The channel priority is set on the default CONDARC setup. Anything created from bob.devtools, by default will have that ON. This allows local reproduction of CI builds.
    • !131 Deps bump: Bumping minor versions of dependencies.
    • !132 Update to latest patch releases: Minor update to try and tackle #46.
    • !133 Update bob/beat-devel to reflect fixes to bob-devel
    • !134 Remove hacks for ripgrep issue (closes #47): Removing hack... Closes #47
    • !135 Resolve "Invalid syntax in setup.py template": Closes #48
    • !136 [bootstrap] use --update-specs during conda install: For some reason when bootstrap installs bob.devtools in the CI of other packages, it does not update all packages. I think this might lead to broken base environments.
    • !137 [bootstrap] specify conda and conda-build versions as specs
    • !138 [bootstrap] add --update-deps in install command
    • !139 Use repodata_from_packages.json for establishing packages to mirror: Use repodata_from_packages.json for establishing package contents on repo to mirror - default channel repodata.json is not updated anymore? See #46
    • !140 Remove /root/.condarc in docker images from continuum: This cleans-up the environment during bootstrap.
    • !141 Allow user-override on channel list: This implements proper user-driven channel override, if one provides a channel list in personal condarc file passed to the command line.
  • v3.0.0
    228eed2d · Increased stable version to 3.0.0 ·
    Release v3.0.0
    • !123 Own implementation of conda channel mirroring: This MR implements a command to perform conda channel mirroring followed by re-indexing. It should replace the use of conda-mirror in your server. Mitigates the remaining issue with #41.
    • !124 Created 2 commands to process gitlab logs: These commands are useful if you want to process gitlab-ci logs. For instance, bdt gitlab get-pipelines bob/bob.nightlies gets all the recent pipelines from bob/bob.nightlies. bdt gitlab process-pipelines bob/bob.nightlies 34567 --job-id 178101 summarizes execution time of a particular pipeline job. If nightlies is set, the stdout is split by project and we have an output like the one below.
    • !112 Bumped conda-build: This MR solves the remaining issue for using conda-build 3.18 (and ripgrep in particular), in our builds. Closes #32
    • !125 Use local proxy: This MR mitigates issues connecting multiple times to https://repo.continuum.io to retrieve the miniconda installer, by using a local reverse proxy caching server.
    • !122 Prepare bob.devtools for .conda package support (c.f. issue #41): This MR prepares our build tools to support .conda packages. Closes #41
    • !127 Reset license file location and remove hack (closes #44): This MR: 1. Removes the hack affecting the builds 2. Corrects the location for the own conda recipe 3. Corrects the location to be used for bdt new runs
    • !126 Implemented a mechanism to run a dependency graph: This MR implements the calculation of dependence graphs for packages which are hosted in gitlab. The command is available via bdt gitlab graph --help. It recursively tries a package resolution via the conda-build API. The user can choose the format of the output, which types of dependencies to track (among host, build, run and test) and what to put on the graph via a regular expression. The user needs access to the gitlab API to use the command. It replaces the old bob_dependency_graph command in bob.extension, that is now removed (see bob.extension!106). Partially addresses #43 (provides the adjancence matrix required for the evaluation of the DAG).
  • v2.0.1
    98b95a10 · Increased stable version to 2.0.1 ·
    Release v2.0.1
    • !108 [bdt new] added bob.extension to templates used for creating new bob.package: This merge request will close #38.
    • !109 Remove conda package caching as we now have a mirror
    • !110 Do not spare downloaded packages from getting cleaned
    • !112 Bumped conda-build: This MR solves the remaining issue for using conda-build 3.18 (and ripgrep in particular), in our builds. Closes #32
    • !113 'info/LICENSE' not in info/files: Trying to solve this issue. bob.nightlies#53
    • !115 Consolidating the hacks implemented to solve conda-build issues
    • !120 This MR addresses the following changes: 1. Update miniconda installer to 4.7.12; 2. Reset installer hashes to avoid Mac re-downloads; 3. Black-ify bootstrap source code
    • !121 Bump bob and beat-devel versions: Implements removal of packaging pin.
  • v2.0.0
    ed35364c · Increased stable version to 2.0.0 ·
    Release v2.0.0
    • !44 Add option to --use-local in build/test: Provide an option to build/test to use local channel for searching packages; Use local packages on nightlies and base builds Closes #26
    • !60 added command to generate documentations locally: This merge request added a new group of commands ("local") to "bdt" command which allows to run the functionalities implemented for CI to run locally. For the moment this group only has "docs" command to generate documents of bob/beat packages locally. fixes partially #19
    • !65 Update matplotlibrc -- turn off latex usage
    • !68 Removed use-local option: It seems the "use-local" is the norm the the new conda-build. This MR removes this option in our builds.
    • !70 [docs-ci] isntalling minimal fonts on the linux docker image: This merge request installs minimal fonts (dejavu-sans-fonts) on the linux docker image so documentations specifically graphviz renders properly.
    • !71 update openfst version
    • !73 New python-gitlab v1.10: Updates python-gitlab to the latest version (v1.10.0)
    • !74 Enable py37 base-builds (bob.conda): This MR does the first step in enabling python 3.7 builds for the core infrastructure (base packages, i.e. those in bob.conda will now be built in both py36 and py37 variants).
    • !75 [bootstrap] Pinning miniconda installer: This MR pins the miniconda installer to circumvent issues with latest installer (see https://gitlab.idiap.ch/bob/bob.devtools/-/jobs/169897) This should be reverted back when #32 is addressed.
    • !76 Update sqlite for py37 compatibility: This improves build compatibility for bob.conda on python-3.7.x
    • !77 [cython] Update to 0.29.12: Update required to recompile menpo/kaldi for python-3.7.x. Closes #36
    • !78 Implements support for specific build-config and recipe-append files: This MR provides a solution to bob.conda#71 by allowing developers to provide branch-specific conda_build_config.yaml and recipe_append.yaml files. All ci scripts and functions will now search for a branch specific files conda_build_config-my-branch.yaml and recipe_append-my-branch.yaml on first on each recipe directory, then on the current directory. If any of those are found, then use these instead of the defaults to build the said package. If they are not found, search for files with names conda_build_config.yaml and recipe_append.yaml on both the recipe dirs, then on the current directory. If any of those are found, they are used. Otherwise, if no specific build variants or recipe-append files are found, use the defaults shipped with this package. Closes bob.conda#71
    • !79 Update bob/beat-devel and sync with available py36/37 builds: This MR updates conda_build_config.yaml to keep both py36 and the new py37 base environments in sync: * Update versions of bob/beat-devel to 2019.07.31 * Update the version of madmom to 0.16.1 * Add mido-1.2.9 (madmom dependence)
    • !80 Update bob devel / pin conda to 4.6: This MR addresses one issue: It brings back menpofit to the list of supported software until second notice. Otherwise, bob.ip.facelandmarks does not build
    • !83 Add a bdt local build command
    • !84 Fix NameErrors in ci base-build and ci test commands: You can see the code failing in https://gitlab.idiap.ch/bob/bob.conda/-/jobs/170227
    • !86 Add black dependence: This MR adds black (code formatter) from the defaults channel as a runtime dependence.
    • !88 [scripts/ci] Fix nightly builds: This MR fixes a copy-n-paste typo on the nightly builds.
    • !89 Docformatter: This MR adds a new dependence to the software stack of bdt, called docformatter. It complements black in re-structuring source code (mostly for docstrings, which get badly formatted with that tool). It also cleans-up unused imports in scripts/local and applies DRY for loading order files. It also applies the results of passing black followed by docformatter on all source code.
    • !90 [conda] Drop free channel from the defaults list: Fixes #33
    • !91 [conda-build-config] Bring back zc_recipe_egg (bob.buildout dep): This MR fixes the nightly builds by bringing back the variable zc_recipe_egg on conda_build_config.yaml.
    • !94 Implement easy WebDAV access: This MR implements a few commands to ease the management of WebDAV resources on our (internal) servers. bdt dav --help should be useful.
    • !97 Check expected miniconda.sh hash, redownloads if necessary: This MR introduces a hash check on the expected version of the miniconda installer. It re-downloads the installer if the hash does not match or if it is not cached. This solves an issue in which more recent versions of the conda (4.7.x) take very long to bootstrap bob.devtools.
    • !99 WebDAV support improvements: This MR adds support for the clean-betas script, which replaces the ad-hoc solution currently installed on our server.
    • !100 More cleanup improvements: This MR: * Hooks the execution of an automatic cleanup action to the successful execution of the build step in nightlies * Provides a manual action to trigger the cleanup. * Makes the clean-betas action a bit more selective on which types of packages it should be cleaning-up (e.g. if running Bob nightlies, do not cleanup BEAT packages, and vice-versa)
    • !101 Changed version of torch and torchvision: Changed version of torch and torchvision
    • !103 Enable py37: This MR enables py37 builds everywhere. It also improves the syntax of CI files to use the new extends keyword instead of YAML templates.
    • !104 Remove oset, add pandas and tqdm: * oset: The package is not used anymore in BEAT. * pandas, tqdm: New requirement for bob.ip.binseg Relates to bob.conda!431
    • !105 [bootstrap] Use local defaults channel mirror
    • !106 Add ncurses for beat/beat.cmdline>: ncurses is needed for the new monitoring command of beat/beat.cmdline> See beat/beat.cmdline!72
  • v1.0.5
    f8d1195a · Increased stable version to 1.0.5 ·
    Release v1.0.5
    • !61 Patched the pattern for the git clean command:
    • !62 Improve remote connection handling and remove alway_yes when developing locally
    • !63 [ci docs] Account for gridtk which has a non-standard name
  • v1.0.4
    b9968bf0 · Increased stable version to 1.0.4 ·
    Release v1.0.4
    • !59 Created the CLI command 'bdt ci docs' to replace the old system which was still depending on shell scripts and bob.admin
  • v1.0.3
    ef4941f3 · Increased stable version to 1.0.3 ·
    Release v1.0.3
    • !53 Update to bob-devel 2019.04.30
    • !54 Update conda_build_config.yaml: Add h5py pinning
    • !55 Adds menpo, menpofit and cyvlfeat
    • !56 Updates jsonschema to 3.0.1 - This MR bumps the version of jsonschema to 3.0.1 on request from beat/beat.core#80. By consequence, it also updates beat-devel to a matching version.
  • v1.0.2
    80e4e3ef · Increased stable version to 1.0.2 ·
    Release v1.0.2
    • !52 Allow for custom conda_build_config.yaml and recipe_append.yaml files: This MR introduces a new feature: if a file named conda_build_config.yaml or recipe_append.yaml exist on the recipe directory, alongside meta.yaml, then they are used to build the package instead of the default files available inside this package.
  • v1.0.1
    9d9cb855 · Increased stable version to 1.0.1 ·
    Release v1.0.1
    • !35 Issue 21: This commit fixes an issue with the upload of documentation for new packages. It ensures the package directory is created on the DAV server before an attempt to upload its documentation.
    • !36 Fix conda/meta.yaml template resolution {{ group }} -> (( group )): This MR fixes an issue when creating new projects. It fixes a place where the template for group was not being taken into consideration. Closes #22.
    • !37 Set macos maximum open file limit to 4096: This MR closes issue #23 and adjusts the docs so that the maximum number of open files on macOS is correctly set from the installation.
    • !38 Build with multiple hdf5 versions at the same time: Update conda_build_config.yaml
    • !39 Remove hdf5 1.10.1: Update conda_build_config.yaml We don't need to build against this anymore.
    • !40 Update dependencies: Depends on bob.conda!409
    • !41 Fix numpy and mkl version incompatibility: Update conda_build_config.yaml This should fix issues in: bob.conda!411
    • !42 Update numpy to earliest version that supports mkl 2019.1: Still trying to fix: bob.conda!411
    • !43 [conda-build-config] update freetype
    • !46 Force re-indexing of locally built packages if "local" channel is present: Apparently, indexes are not refreshed during a tight loop of conda_build.api.build() calls. This MR mitigates it by re-indexing conda-bld at every iteration, if need-be.
    • !45 Requires pyyaml>=5.1, remove deprecation warnings: This MR removes the warning on yaml.load() caused by a deprecation.
    • !47 Avoid deploy deletes, so tight loop package builds work: This MR avoids deploy deletes so that nightlies and bob.conda packages can re-use locally built packages in a tight build loop.
    • !44 Add option to --use-local in build/test: Provide an option to build/test to use local channel for searching packages; Use local packages on nightlies and base builds Closes #26
    • !48 [conda-build-config] Update bob-devel to 2019.04.18: Update conda_build_config.yaml
    • !49 Add PYTHONUNBUFFERED=1 to mitigate #2: This MR sets PYTHONUNBUFFERED=1 everywhere in the CI yaml files. It closes #2. Previous attempts did not work as the interpreter was already started when this option was set.
    • !50 Set bob/bob.devtools#!final for default channels (closes #25): This MR ensures that the defaults_channel list is not changed by conda while being parsed.
    • !51 Avoid stable/master doc deployment from non-master via flag: This MR introduces an option to avoid documentation deployments on the sub-directories master and stable, if the release is not happening from the master branch. It is useful for old patch releases for which one doesn't want to overwrite previous deployments on those directories.
  • v1.0.0
    c758bcfd · Increased stable version to 1.0.0 ·
    Release v1.0.0
    • This is a major release as it marks the official deployment of bdt for managing Bob/BEAT builds in all our packages. Specific changes to the previous release follow:
    • !18 Refines the commitfile command to introduce:
    • Better log messages, more pertinent and with more information about what it is really doing
    • The ability to commit directly to master (which is the default behaviour now)
    • In merge-request creation mode, if [ci skip] or [skip ci] is detected on the commit message, and auto-merge is set, then merge immediately
    • !20 Add support for packages required to run linux builds on docker-build hosts
    • !21 This MR introduces three new features:
    1. A new command bdt test can run test-only conda-builds using tarballs created during a build phase
    2. An equivalent new command bdt ci test can be used to run test-only builds on the CI
    3. A template inside single-package.yaml can be used among packages to run specific test-only builds of themselves
    • !19 [conda-build-config] Update bob/beat-devel versions: Updates bob and beat-devel packages to version 2019.03.07.
    • !23 New doc strategy: Implements and proposes (for new templates) a documentation source installation strategy that makes conda packages testable without the sources (improves previous solution for #5).
    • !24 Allows package namespace to be set and fine-tune conda/documentation urls: Improves the interface of build/test/create to accept a package namespace (group) in order to better tune documentation and conda channel URLs. This fixes a problem currently happening in BEAT package builds in which built documentation is not properly linked. It also improves server URL setup by simplifying it.
    • !26 Introduces the following features:
    1. Consolidate all gitlab-related commands under the command "gitlab"
    2. Add a new command "gitlab runners" to manage runners in projects (enable/disable)
    3. Add a new command "gitlab jobs" to list jobs in shared runners
    • !27 Add missing parameter to create script (closes #15): Closes #15.
    • !28 Nightlies: Centralises nightly builds using bdt infrastructure.
    • !29 Add notes about linux CI setup
    • !25 Do not overwrite conda-build root dir if already configured: The condarc file can contain the root-dir entry which state where conda-build should run and store its artifacts. Don't set croot if that setting is found.
    • !30 Do not set changeps1 to false as that affects deactivation of PS1 on bash: This MR includes the following fixes:
    • Do not set changeps1 to false as that affects deactivation of PS1 in hybrid environments (closes #16)
    • Remove unused channels from default environment
    • Use the name condarc for the rc-file created at the root of each new environment with create (more visible)
    • !31 Add new deployment module and applies DRY to scripts.ci (closes #18): This MR only partially affects issue #18 by encapsulating deployment functionality into two new functions.
    • !32 Update click version to 7.x, click-plugins to 1.0.4
    • !33 No config set when using conda_build.api.build() directly from metadata: This MR implements a change in the use of conda_build.api.build() in the specific cases we are building from the resolved metadata. It reverts using metadata to run the build as that has proved not be reliable. See:
    • !34 Fix use of should_skip_build(): This MR introduces a fix to the use of should_skip_build(). It closes #20 and fixes issues observed at bob.learn.pytorch by @heusch.
  • v0.1.6
    32b079f4 · Increased stable version to 0.1.6 ·
    Release v0.1.6
    • !13 Added conda-verify to avoid a recurring warning during package builds.
    • !14 Implemented resources necessary to build bob/bob.conda packages.
    • !15 Allows the build of base dependencies separating python-independent recipes from python-dependent ones, and potentially using multiple python versions.
    • !16 Introduced a new command called commitfile that is capable to change single files on remote repositories, following a formal merge-request procedure. As an option, it can set auto-merge to true (in case the build succeeds).
    • !17 Introduced the use of terminal-based coloring for better differentiating info, warning and error messages.
  • v0.1.5
    6d0d491b · Increased stable version to 0.1.5 ·
    Release v0.1.5
    • !12 Fixes and properly tests for errors concerning issue #13
  • v0.1.4
    1f49625b · Increased stable version to 0.1.4 ·
    Release v0.1.4
    • !10 Temporarily pin conda-build on 3.16: See bob.conda!398
    • !11 Independent builds and python support: This MR introduces various features: 1. This package is now built completely independently of the bob software stack (and bob.conda). It ships its own version of dependencies and can provide better support for python versions that are not necessarily supported (yet) by Bob or BEAT; 2. As a bonus, you can now install this package alongside your conda base environment; 3. An optimisation during the bootstrap process is now possible since we don't need to install a new environment for bdt using python-3.6, it can just install in the base environment itself; 4. This MR also makes git-clean a bit less verbose, on request form @amohammadi; 5. Finally, it fixes #13, as reported by @heusch
  • v0.1.3
    74c49410 · Increased stable version to 0.1.3 ·
    Release v0.1.3
    • !2 [conda-build-config] Update scikit-image version: This MR updates the file conda_build_config.yaml to the latest version available from bob/bob.admin
    • !4 Implements new project creation closes #3 and #6: This MR introduces the "new" subcommand and implements fixes for bug #3 and #6.
    • !3 [setup] Pin click>=7 and fixes #9: This MR addresses an issue reported by @jdiefenbaugh concerning the build command and click.
    • !5 [bootstrap] Remove usage of conda clean --lock, which does nothing with conda 4.6 anyway: This command is now deprecated in conda and does nothing effectively.
    • !7 [bootstrap] Always use only public channels to bootstrap (closes #11): This MR addresses an issue in which, for private packages, the CI will tend to install bdt from the private beta channel instead of the latest version from the public beta channel, which should be the case.
    • !6 [config] Pin click at different location: Actually conda_build_config.yaml has nothing to-do with the dependencies of bdt. So, pin click>=7 only for this package.
    • !9 Add options to allow testing local builds: This will make local building (for testing purposes) work correctly.
    • !8 This MR suggests a configuration change for macOS-based builders (shared) to avoid cross-talking between miniconda installations. Closes #12.