Bumped conda-build
This MR solves the remaining issue for using conda-build 3.18 (and ripgrep in particular), in our builds.
Closes #32 (closed)
Merge request reports
Activity
Guys, I don't know what's going on.
Follow bellow the steps to reproduce the issue using
conda-build
directly and thecontinuumio
image$ docker run -it continuumio/conda-concourse-ci /bin/bash $ git clone https://gitlab.idiap.ch/bob/bob.devtools $ git checkout -b bump-conda-build origin/bump-conda-build $ cd bob.devtools $ vi ./bob/bob.devtools/bootstrap.py +445 # Replace this line to this: f.write(_BASE_CONDARC) $ python3 ./bob/devtools/bootstrap.py -vv build $ source ./miniconda/etc/profile.d/conda.sh $ conda activate base $ conda config --add channels https://idiap.ch/software/bob/conda $ conda build -m ./bob/devtools/data/conda_build_config.yaml ./conda/
It seems an issue with our recipe.
I tried to create an isolated project (with nothing inside) just to see if I can observe the issue, but everything works fine.
Follow the steps to reproduce the success:
git clone https://github.com/tiagofrepereira2012/dummy cd dummy wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh chmod +x Miniconda3-4.7.10-Linux-x86_64.sh ./Miniconda3-4.7.10-Linux-x86_64.sh -b -p ./miniconda2 # Install on ./miniconda/ source ./miniconda2/etc/profile.d/conda.sh conda activate base conda install conda-build=3.18 --yes conda build --no-anaconda-upload -m build_config.yaml ./conda/
Ideas??
ping @andre.anjos @amohammadi
Edited by Tiago de Freitas PereiraThe test script in "https://github.com/tiagofrepereira2012/dummy/blob/master/conda/meta.yaml#L27" runs fine so we can maybe exclude the
charset
hypothesisIt works :-(
I added everything https://github.com/tiagofrepereira2012/dummy/blob/master/conda/meta.yaml
Follow the log of my dummy test. As you can see, hard-coded paths are found and replaced
(base) [root@e4dae9910f9e dummy]# conda build --no-anaconda-upload -m build_config.yaml ./conda/ No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11 WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11 Adding in variants from internal_defaults INFO:conda_build.variants:Adding in variants from internal_defaults Adding in variants from build_config.yaml INFO:conda_build.variants:Adding in variants from build_config.yaml Attempting to finalize metadata for dummy INFO:conda_build.metadata:Attempting to finalize metadata for dummy Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done BUILD START: [u'dummy-0.0.1-py36he61d863_0.tar.bz2'] Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done ## Package Plan ## environment location: /build_scripts/miniconda/conda-bld/dummy_1570605239348/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main defaults ca-certificates: 2019.8.28-0 defaults certifi: 2019.9.11-py36_0 defaults libedit: 3.1.20181209-hc058e9b_0 defaults libffi: 3.2.1-hd88cf55_4 defaults libgcc-ng: 9.1.0-hdf63c60_0 defaults libstdcxx-ng: 9.1.0-hdf63c60_0 defaults ncurses: 6.1-he6710b0_1 defaults openssl: 1.1.1d-h7b6447c_2 defaults python: 3.6.9-h265db76_0 defaults readline: 7.0-h7b6447c_5 defaults setuptools: 41.4.0-py36_0 defaults sqlite: 3.30.0-h7b6447c_0 defaults tk: 8.6.8-hbc83047_0 defaults xz: 5.2.4-h14c3975_4 defaults zlib: 1.2.11-h7b6447c_3 defaults Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done source tree in: /build_scripts/miniconda/conda-bld/dummy_1570605239348/work export SRC_DIR=/build_scripts/miniconda/conda-bld/dummy_1570605239348/work export PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla export BUILD_PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_build_env running sdist running egg_info creating dummy.egg-info writing dummy.egg-info/PKG-INFO writing dependency_links to dummy.egg-info/dependency_links.txt writing entry points to dummy.egg-info/entry_points.txt writing top-level names to dummy.egg-info/top_level.txt writing manifest file 'dummy.egg-info/SOURCES.txt' reading manifest file 'dummy.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'dummy.egg-info/SOURCES.txt' running check creating dummy-0.0.1 creating dummy-0.0.1/dummy creating dummy-0.0.1/dummy.egg-info copying files to dummy-0.0.1... copying MANIFEST.in -> dummy-0.0.1 copying README.md -> dummy-0.0.1 copying setup.py -> dummy-0.0.1 copying dummy/__init__.py -> dummy-0.0.1/dummy copying dummy/dummy.py -> dummy-0.0.1/dummy copying dummy.egg-info/PKG-INFO -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/SOURCES.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/dependency_links.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/entry_points.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/top_level.txt -> dummy-0.0.1/dummy.egg-info Writing dummy-0.0.1/setup.cfg creating dist creating 'dist/dummy-0.0.1.zip' and adding 'dummy-0.0.1' to it adding 'dummy-0.0.1' adding 'dummy-0.0.1/dummy.egg-info' adding 'dummy-0.0.1/dummy' adding 'dummy-0.0.1/PKG-INFO' adding 'dummy-0.0.1/setup.py' adding 'dummy-0.0.1/README.md' adding 'dummy-0.0.1/setup.cfg' adding 'dummy-0.0.1/MANIFEST.in' adding 'dummy-0.0.1/dummy.egg-info/PKG-INFO' adding 'dummy-0.0.1/dummy.egg-info/SOURCES.txt' adding 'dummy-0.0.1/dummy.egg-info/entry_points.txt' adding 'dummy-0.0.1/dummy.egg-info/top_level.txt' adding 'dummy-0.0.1/dummy.egg-info/dependency_links.txt' adding 'dummy-0.0.1/dummy/__init__.py' adding 'dummy-0.0.1/dummy/dummy.py' removing 'dummy-0.0.1' (and everything under it) running install running build running build_py creating build creating build/lib creating build/lib/dummy copying dummy/__init__.py -> build/lib/dummy copying dummy/dummy.py -> build/lib/dummy running install_lib creating $PREFIX/lib/python3.6/site-packages/dummy copying build/lib/dummy/__init__.py -> $PREFIX/lib/python3.6/site-packages/dummy copying build/lib/dummy/dummy.py -> $PREFIX/lib/python3.6/site-packages/dummy byte-compiling $PREFIX/lib/python3.6/site-packages/dummy/__init__.py to __init__.cpython-36.pyc byte-compiling $PREFIX/lib/python3.6/site-packages/dummy/dummy.py to dummy.cpython-36.pyc running install_egg_info running egg_info writing dummy.egg-info/PKG-INFO writing dependency_links to dummy.egg-info/dependency_links.txt writing entry points to dummy.egg-info/entry_points.txt writing top-level names to dummy.egg-info/top_level.txt reading manifest file 'dummy.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'dummy.egg-info/SOURCES.txt' Copying dummy.egg-info to $PREFIX/lib/python3.6/site-packages/dummy-0.0.1-py3.6.egg-info running install_scripts Installing dummy_print.py script to $PREFIX/bin writing list of installed files to 'record.txt' Resource usage statistics from building dummy: Process count: 1 CPU time: Sys=0:00:00.0, User=0:00:00.0 Memory: 2.8M Disk usage: 20B Time elapsed: 0:00:02.1 Packaging dummy INFO:conda_build.build:Packaging dummy INFO conda_build.build:build(1570): Packaging dummy Packaging dummy-0.0.1-py36he61d863_0 INFO:conda_build.build:Packaging dummy-0.0.1-py36he61d863_0 INFO conda_build.build:bundle_conda(891): Packaging dummy-0.0.1-py36he61d863_0 compiling .pyc files... number of files: 10 Fixing permissions Detected hard-coded path in text file bin/dummy_print.py Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away. WARNING:conda_build.build:Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away. WARNING conda_build.build:bundle_conda(1042): Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away. TEST START: /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py36he61d863_0.tar.bz2 Renaming work directory, /build_scripts/miniconda/conda-bld/dummy_1570605239348/work to /build_scripts/miniconda/conda-bld/dummy_1570605239348/work_moved_dummy-0.0.1-py36he61d863_0_linux-64 Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done ## Package Plan ## environment location: /build_scripts/miniconda/conda-bld/dummy_1570605239348/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main defaults ca-certificates: 2019.8.28-0 defaults certifi: 2019.9.11-py36_0 defaults dummy: 0.0.1-py36he61d863_0 local libedit: 3.1.20181209-hc058e9b_0 defaults libffi: 3.2.1-hd88cf55_4 defaults libgcc-ng: 9.1.0-hdf63c60_0 defaults libstdcxx-ng: 9.1.0-hdf63c60_0 defaults ncurses: 6.1-he6710b0_1 defaults openssl: 1.1.1d-h7b6447c_2 defaults python: 3.6.9-h265db76_0 defaults readline: 7.0-h7b6447c_5 defaults setuptools: 41.4.0-py36_0 defaults sqlite: 3.30.0-h7b6447c_0 defaults tk: 8.6.8-hbc83047_0 defaults xz: 5.2.4-h14c3975_4 defaults zlib: 1.2.11-h7b6447c_3 defaults Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done export SRC_DIR=/build_scripts/miniconda/conda-bld/dummy_1570605239348/test_tmp export PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol import: 'dummy' import: 'dummy' + dummy_print.py Dummy dummy dummy dummy !!!! + exit 0 Resource usage statistics from testing dummy: Process count: 1 CPU time: unavailable Memory: 8.0K Disk usage: 20B Time elapsed: 0:00:02.1 TEST END: /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py36he61d863_0.tar.bz2 Renaming work directory, /build_scripts/miniconda/conda-bld/dummy_1570605239348/work to /build_scripts/miniconda/conda-bld/dummy_1570605239348/work_moved_dummy-0.0.1-py36he61d863_0_linux-64_main_build_loop Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done BUILD START: [u'dummy-0.0.1-py37he61d863_0.tar.bz2'] Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done ## Package Plan ## environment location: /build_scripts/miniconda/conda-bld/dummy_1570605239348/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main defaults ca-certificates: 2019.8.28-0 defaults certifi: 2019.9.11-py37_0 defaults libedit: 3.1.20181209-hc058e9b_0 defaults libffi: 3.2.1-hd88cf55_4 defaults libgcc-ng: 9.1.0-hdf63c60_0 defaults libstdcxx-ng: 9.1.0-hdf63c60_0 defaults ncurses: 6.1-he6710b0_1 defaults openssl: 1.1.1d-h7b6447c_2 defaults python: 3.7.4-h265db76_1 defaults readline: 7.0-h7b6447c_5 defaults setuptools: 41.4.0-py37_0 defaults sqlite: 3.30.0-h7b6447c_0 defaults tk: 8.6.8-hbc83047_0 defaults xz: 5.2.4-h14c3975_4 defaults zlib: 1.2.11-h7b6447c_3 defaults Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done source tree in: /build_scripts/miniconda/conda-bld/dummy_1570605239348/work export SRC_DIR=/build_scripts/miniconda/conda-bld/dummy_1570605239348/work export PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla export BUILD_PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_build_env running sdist running egg_info writing dummy.egg-info/PKG-INFO writing dependency_links to dummy.egg-info/dependency_links.txt writing entry points to dummy.egg-info/entry_points.txt writing top-level names to dummy.egg-info/top_level.txt reading manifest file 'dummy.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'dummy.egg-info/SOURCES.txt' running check creating dummy-0.0.1 creating dummy-0.0.1/dummy creating dummy-0.0.1/dummy.egg-info copying files to dummy-0.0.1... copying MANIFEST.in -> dummy-0.0.1 copying README.md -> dummy-0.0.1 copying setup.py -> dummy-0.0.1 copying dummy/__init__.py -> dummy-0.0.1/dummy copying dummy/dummy.py -> dummy-0.0.1/dummy copying dummy.egg-info/PKG-INFO -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/SOURCES.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/dependency_links.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/entry_points.txt -> dummy-0.0.1/dummy.egg-info copying dummy.egg-info/top_level.txt -> dummy-0.0.1/dummy.egg-info Writing dummy-0.0.1/setup.cfg creating 'dist/dummy-0.0.1.zip' and adding 'dummy-0.0.1' to it adding 'dummy-0.0.1' adding 'dummy-0.0.1/dummy.egg-info' adding 'dummy-0.0.1/dummy' adding 'dummy-0.0.1/PKG-INFO' adding 'dummy-0.0.1/setup.py' adding 'dummy-0.0.1/README.md' adding 'dummy-0.0.1/setup.cfg' adding 'dummy-0.0.1/MANIFEST.in' adding 'dummy-0.0.1/dummy.egg-info/PKG-INFO' adding 'dummy-0.0.1/dummy.egg-info/SOURCES.txt' adding 'dummy-0.0.1/dummy.egg-info/entry_points.txt' adding 'dummy-0.0.1/dummy.egg-info/top_level.txt' adding 'dummy-0.0.1/dummy.egg-info/dependency_links.txt' adding 'dummy-0.0.1/dummy/__init__.py' adding 'dummy-0.0.1/dummy/dummy.py' removing 'dummy-0.0.1' (and everything under it) running install running build running build_py running install_lib creating $PREFIX/lib/python3.7/site-packages/dummy copying build/lib/dummy/__init__.py -> $PREFIX/lib/python3.7/site-packages/dummy copying build/lib/dummy/dummy.py -> $PREFIX/lib/python3.7/site-packages/dummy byte-compiling $PREFIX/lib/python3.7/site-packages/dummy/__init__.py to __init__.cpython-37.pyc byte-compiling $PREFIX/lib/python3.7/site-packages/dummy/dummy.py to dummy.cpython-37.pyc running install_egg_info running egg_info writing dummy.egg-info/PKG-INFO writing dependency_links to dummy.egg-info/dependency_links.txt writing entry points to dummy.egg-info/entry_points.txt writing top-level names to dummy.egg-info/top_level.txt reading manifest file 'dummy.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'dummy.egg-info/SOURCES.txt' Copying dummy.egg-info to $PREFIX/lib/python3.7/site-packages/dummy-0.0.1-py3.7.egg-info running install_scripts Installing dummy_print.py script to $PREFIX/bin writing list of installed files to 'record.txt' Resource usage statistics from building dummy: Process count: 1 CPU time: unavailable Memory: 688.0K Disk usage: 20B Time elapsed: 0:00:02.1 compiling .pyc files... number of files: 10 Fixing permissions Detected hard-coded path in text file bin/dummy_print.py TEST START: /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py37he61d863_0.tar.bz2 Renaming work directory, /build_scripts/miniconda/conda-bld/dummy_1570605239348/work to /build_scripts/miniconda/conda-bld/dummy_1570605239348/work_moved_dummy-0.0.1-py37he61d863_0_linux-64 Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done ## Package Plan ## environment location: /build_scripts/miniconda/conda-bld/dummy_1570605239348/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main defaults ca-certificates: 2019.8.28-0 defaults certifi: 2019.9.11-py37_0 defaults dummy: 0.0.1-py37he61d863_0 local libedit: 3.1.20181209-hc058e9b_0 defaults libffi: 3.2.1-hd88cf55_4 defaults libgcc-ng: 9.1.0-hdf63c60_0 defaults libstdcxx-ng: 9.1.0-hdf63c60_0 defaults ncurses: 6.1-he6710b0_1 defaults openssl: 1.1.1d-h7b6447c_2 defaults python: 3.7.4-h265db76_1 defaults readline: 7.0-h7b6447c_5 defaults setuptools: 41.4.0-py37_0 defaults sqlite: 3.30.0-h7b6447c_0 defaults tk: 8.6.8-hbc83047_0 defaults xz: 5.2.4-h14c3975_4 defaults zlib: 1.2.11-h7b6447c_3 defaults Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done export SRC_DIR=/build_scripts/miniconda/conda-bld/dummy_1570605239348/test_tmp export PREFIX=/build_scripts/miniconda/conda-bld/dummy_1570605239348/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol import: 'dummy' import: 'dummy' + dummy_print.py Dummy dummy dummy dummy !!!! + exit 0 Resource usage statistics from testing dummy: Process count: 1 CPU time: Sys=0:00:00.0, User=0:00:00.0 Memory: 8.0K Disk usage: 20B Time elapsed: 0:00:02.1 TEST END: /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py37he61d863_0.tar.bz2 Renaming work directory, /build_scripts/miniconda/conda-bld/dummy_1570605239348/work to /build_scripts/miniconda/conda-bld/dummy_1570605239348/work_moved_dummy-0.0.1-py37he61d863_0_linux-64_main_build_loop # Automatic uploading is disabled # If you want to upload package(s) to anaconda.org later, type: anaconda upload /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py36he61d863_0.tar.bz2 anaconda upload /build_scripts/miniconda/conda-bld/linux-64/dummy-0.0.1-py37he61d863_0.tar.bz2 # To have conda build upload to anaconda.org automatically, use # $ conda config --set anaconda_upload yes anaconda_upload is not set. Not uploading wheels: [] #################################################################################### Resource usage summary: Total time: 0:01:26.9 CPU usage: sys=0:00:00.0, user=0:00:00.0 Maximum memory usage observed: 2.8M Total disk usage observed (not including envs): 60B
@andre.anjos, what's so special in your bootstrap script?
This is true for all bob packages. For instance, if you try to
conda-build
bob.extension
is fails tooFollow bellow the lines to reproduce
git clone https://gitlab.idiap.ch/bob/bob.extension && cd bob.extension && wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh && chmod +x Miniconda3-4.7.10-Linux-x86_64.sh && ./Miniconda3-4.7.10-Linux-x86_64.sh -b -p ./miniconda2 && source ./miniconda2/etc/profile.d/conda.sh && conda activate base && conda install -n base python conda-build=3 conda=4 conda-verify=3 --yes -c https://idiap.ch/software/bob/conda && wget https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/data/conda_build_config.yaml && conda build --no-anaconda-upload -c https://idiap.ch/software/bob/conda -m conda_build_config.yaml ./conda/
The problem is that
ripgrep
, a new dependence to conda-build (introduced in 3.18), thinks it is a good idea to use VCS ignore files (such as.gitignore
) to ignore part of the files to be searched. This probably makes sense in the context of a project, but less in the context of a conda build. Because we re-install miniconda within our checkouts at the CI, that contain an entry for ignoringbin
directories, the end result is that ripgrep picks that ignore file and thinks it needs to ignore thebin/
directory on the installation it is searching for prefix matches. Because all of the scripts go the bin directory, hard-coded paths are not detected, which causes the above error.The fix:
-
@tiago.pereira will report an issue in the conda-build bug tracker with a fix (add
--no-ignore
to ripgrep's command line within conda-build) - Meanwhile, we will patch our CI builds so that we remove the
.gitignore
of the current checkout to avoid this problem.
-
@tiago.pereira will report an issue in the conda-build bug tracker with a fix (add
mentioned in commit 03dfce90
added 1 commit
- 03dfce90 - [bootstrap] Remove .gitignore file from current checkout to avoid issues with ripgrep (see !112 (merged))
assigned to @andre.anjos
mentioned in commit c99c3336
I opened a merge request here: https://github.com/conda/conda-build/pull/3766
Follow the steps to reproduce the issue
git clone https://github.com/tiagofrepereira2012/dummy && \ cd dummy && \ wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh && \ bash ./Miniconda3-4.7.10-Linux-x86_64.sh -b -p ./miniconda && \ source ./miniconda/etc/profile.d/conda.sh && \ conda activate base && \ conda install -n base python conda-build=3 conda=4 conda-verify=3 --yes && \ conda build ./conda --no-anaconda-upload -m build_config.yaml
Just an update.
https://github.com/conda/conda-build/pull/3766
Very soon we'll be able to remove one of our hacks