Buildout takes forever to finish on our conda envs (even longer on idiap machines)
It takes 29 seconds on my laptop which has NVME SSD and core i7 pro to do a simple buildout. I did a profiling on my laptop:
amir@archie ~/idiap/git/bobs/bob.extension (git)-[master] % buildout
Creating directory '/home/amir/idiap/git/bobs/bob.extension/eggs'.
Creating directory '/home/amir/idiap/git/bobs/bob.extension/bin'.
Creating directory '/home/amir/idiap/git/bobs/bob.extension/parts'.
Creating directory '/home/amir/idiap/git/bobs/bob.extension/develop-eggs'.
Develop: '/home/amir/idiap/git/bobs/bob.extension/.'
Getting distribution for 'bob.buildout'.
Got bob.buildout 2.1.0.
Getting distribution for 'zc.recipe.egg>=2.0.0a3'.
Got zc.recipe.egg 2.0.3.
Installing scripts.
Python: Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/python'.
GdbPython: Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/gdb-python'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/bob_dependecy_graph.py'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/bob_new_version.py'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/nosetests'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/coverage'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/sphinx-apidoc'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/sphinx-build'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/sphinx-quickstart'.
Generated script '/home/amir/idiap/git/bobs/bob.extension/bin/sphinx-autogen'.
buildout 25,80s user 2,22s system 95% cpu 29,398 total
amir@archie ~/idiap/git/bobs/bob.extension (git)-[master] % ipython
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import pstats
...: p = pstats.Stats('results.txt')
...: p.strip_dirs().sort_stats(2).print_stats()
Tue Mar 14 17:49:57 2017 results.txt
13420083 function calls (13389173 primitive calls) in 24.035 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 24.083 24.083 buildout:2(<module>)
3 0.001 0.000 15.669 5.223 easy_install.py:894(install)
1 0.001 0.001 12.678 12.678 buildout.py:15(<module>)
1 0.034 0.034 12.619 12.619 easy_install.py:19(<module>)
11 0.110 0.010 12.410 1.128 __init__.py:1068(scan)
9 0.000 0.000 12.409 1.379 __init__.py:1030(__init__)
3 0.000 0.000 12.289 4.096 easy_install.py:229(__init__)
3 0.000 0.000 12.234 4.078 easy_install.py:273(_make_env)
1 0.000 0.000 11.404 11.404 buildout.py:1889(main)
1 0.000 0.000 11.400 11.400 buildout.py:506(install)
55655/54745 0.366 0.000 9.814 0.000 __init__.py:1967(find_on_path)
54376 0.349 0.000 8.730 0.000 __init__.py:2385(from_location)
1 0.000 0.000 6.484 6.484 buildout.py:805(_compute_part_signatures)
1 0.000 0.000 6.482 6.482 buildout.py:1686(_dists_sig)
2 0.123 0.061 6.482 3.241 buildout.py:1650(_dir_hash)
51664 0.111 0.000 5.801 0.000 __init__.py:2728(_reload_version)
51664 0.142 0.000 5.534 0.000 __init__.py:2357(_version_from_file)
157562 1.504 0.000 3.797 0.000 version.py:198(__init__)
51664 0.088 0.000 3.607 0.000 {next}
207074 0.277 0.000 3.516 0.000 __init__.py:2530(_get_metadata)
57499 3.502 0.000 3.502 0.000 {method 'read' of 'file' objects}
106089 0.258 0.000 3.485 0.000 __init__.py:1381(safe_version)
138/96 0.000 0.000 3.404 0.035 buildout.py:1126(__getitem__)
3 0.000 0.000 3.404 1.135 buildout.py:1228(_initialize)
1 0.000 0.000 3.404 3.404 buildout.py:1251(initialize)
1 0.000 0.000 3.403 3.403 buildout.py:1180(_install_and_load)
3 0.000 0.000 3.379 1.126 easy_install.py:648(install)
4 0.000 0.000 3.235 0.809 easy_install.py:527(_get_dist)
51212 0.207 0.000 3.227 0.000 __init__.py:1094(add)
20 0.000 0.000 3.082 0.154 subprocess.py:160(call)
40 0.000 0.000 3.066 0.077 subprocess.py:118(_eintr_retry_call)
20 0.000 0.000 3.037 0.152 subprocess.py:1068(wait)
20 3.036 0.152 3.036 0.152 {posix.waitpid}
50375 0.128 0.000 2.543 0.000 __init__.py:2431(__eq__)
2 0.000 0.000 2.426 1.213 easy_install.py:1662(_move_to_eggs_dir_and_compile)
54460 0.202 0.000 2.249 0.000 __init__.py:2373(__init__)
102633 0.186 0.000 2.242 0.000 __init__.py:2405(hashcmp)
2 0.000 0.000 2.136 1.068 easy_install.py:1579(call_easy_install)
27620 1.842 0.000 1.842 0.000 {method 'update' of '_hashlib.HASH' objects}
102688 0.114 0.000 1.480 0.000 __init__.py:2452(parsed_version)
51189 0.053 0.000 1.220 0.000 __init__.py:204(parse_version)
18350 0.050 0.000 1.123 0.000 __init__.py:1872(get_metadata_lines)
18350 0.180 0.000 1.074 0.000 __init__.py:1859(get_metadata)
18 0.000 0.000 0.947 0.053 easy_install.py:154(call_subprocess)
207656/207151 0.268 0.000 0.912 0.000 __init__.py:2193(yield_lines)
157237 0.780 0.000 0.868 0.000 version.py:343(_cmpkey)
1 0.000 0.000 0.829 0.829 buildout.py:1413(_call)
1 0.000 0.000 0.829 0.829 scripts.py:237(install)
180 0.005 0.000 0.790 0.004 __init__.py:688(add_entry)
6 0.001 0.000 0.745 0.124 script.py:110(working_set)
20357/3463 0.075 0.000 0.744 0.000 os.py:209(walk)