Bumped conda-build

@tiago.pereira: There is more than that to change... You must also edit the bootstrap.py file.

yeap, i know.

I've done it here b5386fb7

However there's an issue that i'm debugging

marked as a Work In Progress

assigned to @andre.anjos

added 1 commit

37352843 - Bumped conda-build

Compare with previous version

changed the description

mentioned in issue #32 (closed)

I can reproduce it locally.

INFO conda_build.build:build(1570): Packaging bob.devtools
Packaging bob.devtools
/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/environ.py:426: UserWarning: The environment variable 'DOCSERVER' is being passed through with value http://www.idiap.ch.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
Packaging bob.devtools-2.0.1b0-py37h661208a_3
INFO conda_build.build:bundle_conda(891): Packaging bob.devtools-2.0.1b0-py37h661208a_3
compiling .pyc files...
  File "lib/python3.7/site-packages/bob/devtools/templates/setup.py", line 4
    {% if group == 'beat' %}from setuptools import setup, find_packages
     ^
SyntaxError: invalid syntax

number of files: 141
Fixing permissions
Packaged license file/s.
'info/LICENSE' not in info/files
Traceback (most recent call last):
  File "./bob/devtools/build.py", line 820, in <module>
    conda_build.api.build(recipe_dir, config=conda_config)
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/api.py", line 209, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 2343, in build_tree
    notest=notest,
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1642, in build
    newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1034, in bundle_conda
    tarcheck.check_all(tmp_path, metadata.config)
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/tarcheck.py", line 89, in check_all
    x.info_files()
  File "/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/tarcheck.py", line 53, in info_files
    raise Exception('info/files')
Exception: info/files
ERROR: Job failed: exit code 1

It seems we need an info/files listing all files that are part of the package itself.

https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html

@tiago.pereira: that link specifies what goes within an already built conda package. Not how to write conda/meta.yaml so that it happens.

I should also add that the LICENSE file was specified "correctly" in conda/meta.yaml. So, my understanding is that it should have ended there. What is wrong?

don't know.

i'm debugging here

This condition is not holding

https://github.com/conda/conda-build/blob/3.18.9/conda_build/tarcheck.py#L46

info/LICENSE is missing from seta

if you look at the log the LICENSE file is not being copied

in short the info/LICENSE is on the tarball (setb in the code above), but not in the info/files (seta)

Can you check what happens if:

You say the file is "LICENSE" instead of "../LICENSE", or
If you remove that entry completely from meta.yaml?

If you remove it from the meta.yml, it works

if you do license: LICENSE instead of license: ../LICENSE, it breaks

I can't find a similar problem in anaconda issues.

Did you find it something?

Here: https://github.com/conda/conda-build/issues/2657

FYI, searched for "not in info/files" conda

thanks,

wooooow, there are 654 opened issues in conda-build

To make it work AND ship the license, we could create a symlink in conda dir

@amohammadi: shall we continue to pin all conda dependencies like this, or could we git rid of this pinning as well?

These are very relaxed pins and stop our builds form breaking if a next major version is released. Why do you want to remove them?

changed this line in version 11 of the diff

added 1 commit

b9beb0c9 - Work around to conda-build work and ship the LICENSE

Compare with previous version

added 1 commit

512e0a2a - Work around to conda-build work and ship the LICENSE

Compare with previous version

added 1 commit

6a30f46d - Work around to conda-build work and ship the LICENSE

Compare with previous version

added 1 commit

45e2eff8 - Work around to conda-build work and ship the LICENSE

Compare with previous version

The doc says the path to the license file is relative to the source directory. However, when you say:

if you do license: LICENSE instead of license: ../LICENSE, it breaks

It looks like the path is relative to the recipe directory?!

Doesn't this look like a conda build bug?

Now the issue is another thing.

/scratch/builds/bob/bob.devtools/miniconda/lib/python3.7/site-packages/conda_build/environ.py:426: UserWarning: The environment variable 'DOCSERVER' is being passed through with value http://www.idiap.ch.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
export PREFIX=/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
export SRC_DIR=/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/test_tmp
import: 'bob.devtools'
import: 'bob.devtools'
+ bdt -h
/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/test_tmp/run_test.sh: /scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/bdt: /scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_157008995652: bad interpreter: No such file or directory
Tests failed for bob.devtools-2.0.1b0-py36h661208a_3.tar.bz2 - moving package to /scratch/builds/bob/bob.devtools/miniconda/conda-bld/broken
WARNING conda_build.build:tests_failed(2230): Tests failed for bob.devtools-2.0.1b0-py36h661208a_3.tar.bz2 - moving package to /scratch/builds/bob/bob.devtools/miniconda/conda-bld/broken
TESTS FAILED: bob.devtools-2.0.1b0-py36h661208a_3.tar.bz2

The python interpreter set in /scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/bdt

does not exist

it is pointing to something like /scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/_h_***** instead of ``/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089956525/_test_env_placehold_`

any ideas?

How this thing can be so buggy with minor bumps?

we should move to matlab

mentioned in merge request beat/beat.core!99 (merged)

I need some light on this, I would be very thankful if someone has an idea on what's going on.

Cheers

For some reason, at some point of the build, the file /scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1570089934535/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol is deleted

@tiago.pereira most probably you are hitting a conda build bug. Try downgrading conda build and find out at which version of conda-build it stops working. Then, report it as an issue on conda-build.

@tiago.pereira I can build this locally without an error. Here is my conda info:

conda info                                                                                                                                                                     :(

     active environment : base
    active env location : /home/amir/miniconda
            shell level : 1
       user config file : /home/amir/.condarc
 populated config files : /home/amir/miniconda/.condarc
                          /home/amir/.condarc
          conda version : 4.7.12
    conda-build version : 3.18.9
         python version : 3.6.9.final.0
       virtual packages : 
       base environment : /home/amir/miniconda  (writable)
           channel URLs : https://www.idiap.ch/software/bob/conda/linux-64
                          https://www.idiap.ch/software/bob/conda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/amir/miniconda/pkgs
                          /home/amir/.conda/pkgs
       envs directories : /home/amir/miniconda/envs
                          /home/amir/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.6.9 Linux/5.3.1-arch1-1-ARCH arch/rolling glibc/2.29
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

and here is what I ran:

cd bob.devtools
conda build -m bob/devtools/data/conda_build_config.yaml conda

is this the master branch?

yes. let me try this branch too.

My container

conda info

     active environment : base
    active env location : /build_scripts/bob.devtools/miniconda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /build_scripts/bob.devtools/miniconda/condarc
                          /root/.condarc
          conda version : 4.7.12
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages : 
       base environment : /build_scripts/bob.devtools/miniconda  (writable)
           channel URLs : https://conda.anaconda.org/c3i_test/linux-64
                          https://conda.anaconda.org/c3i_test/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
          package cache : /build_scripts/bob.devtools/miniconda/pkgs
                          /root/.conda/pkgs
       envs directories : /build_scripts/bob.devtools/miniconda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.12 requests/2.21.0 CPython/3.7.4 Linux/4.9.125-linuxkit centos/6.9 glibc/2.12
                UID:GID : 0:0
             netrc file : None
           offline mode : False

@tiago.pereira this is what I added on the top of this branch:

diff --git a/conda/meta.yaml b/conda/meta.yaml
index 059b565..14b3b13 100644
--- a/conda/meta.yaml
+++ b/conda/meta.yaml
@@ -48,9 +48,6 @@ requirements:
     - jinja2
     - termcolor
     - psutil
-    # pin packaging temporarily until
-    # https://github.com/ContinuumIO/anaconda-issues/issues/11236 is fixed.
-    - packaging=19.0
 
 test:
   requires:

and this is what I run:

conda build -m bob/devtools/data/conda_build_config.yaml conda

It worked for both py3.6 and py3.7

@tiago.pereira why do you have the testing channels in your list? https://conda.anaconda.org/c3i_test

I have no idea, but this is the norm since I don't know when.

What is the issue?

The issue is that we shouldn't be using that. Those are test channels where beta versions of conda-build and other packages get uploaded there.

Downgrading to conda-build 3.17 it works

I need to leave. Have no idea what's going on. I can't make it work with conda-build>=3.18

added 1 commit

8fd9915f - Try a fix

Compare with previous version

added 1 commit

5d2e572b - skip compilation of template files

Compare with previous version

added 1 commit

1c13152c - see what is inside bin/bdt

Compare with previous version

added 1 commit

b706c45e - see what is inside bin/bdt

Compare with previous version

@tiago.pereira I added some test code to see what goes inside bin/bdt:

#!$PREFIX/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'bob.devtools==2.0.1b0','console_scripts','bdt'
__requires__ = 'bob.devtools==2.0.1b0'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('bob.devtools==2.0.1b0', 'console_scripts', 'bdt')()
    )

This definitely looks like a conda-build bug. Please report it upstream if you have the time.

added 1 commit

33dfa1cc - Pinning conda-build 3.17

Compare with previous version

Guys, I pinned conda-build to version 3.17

I would like to see if this will solve the issue from https://gitlab.idiap.ch/bob/bob.nightlies/issues/52

I'll open a new merge request to remove this pin as soon as we figure out what's going on

unmarked as a Work In Progress

changed the description

merged

mentioned in commit 5606e84b

mentioned in merge request !112 (merged)

Bumped conda-build

Merge request reports

Activity