Cannot see package tarballs in the browser

When I visit: https://www.idiap.ch/software/bob/conda/linux-64/ or any other folder in our channel. I can only see 20 packages or so. However, when I do conda search and conda install all packages are there!

Anyone knows why?

Possibly related to the Apache configuration. Was it changed or updated?

Hey @amohammadi, you were playing with the apache yesterday, no?

no I was changing the conda installation there.

This may be due to an update of the package (or lack of). I'm running an upgrade to see if that improves things.

Nevertheless, it looks super-fishy.

Here is the problem.

Apparently, the apache server is picking up a file that is autogenerated by conda indexing called channeldata.json.

This file for the root of the conda channel is this (grabbed with cat /local/web/public/conda/channeldata.json as www-data user on our server):

$ cat /local/web/public/conda/channeldata.json
{
  "channeldata_version": 1,
  "packages": {
    "bob-extras": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "https://www.idiap.ch/software/bob/",
      "license": "BSD License",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/bob-extras-2017.10.22-0.tar.bz2",
      "run_exports": {},
      "subdirs": [
        "linux-64",
        "osx-64"
      ],
      "summary": "A meta package which pins down all extra packages.",
      "text_prefix": false,
      "version": "2017.10.22"
    },
    "eigen": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "http://eigen.tuxfamily.org/",
      "license": "MPL2",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/eigen-3.3.1-3.tar.bz2",
      "run_exports": {},
      "source_url": "http://bitbucket.org/eigen/eigen/get/3.3.1.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "C++ template library for linear algebra",
      "text_prefix": true,
      "version": "3.3.1"
    },
    "giflib": {
      "activate.d": false,
      "binary_prefix": true,
      "deactivate.d": false,
      "home": "http://giflib.sourceforge.net",
      "license": "MIT",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/giflib-5.1.4-h9a61bd3_4.tar.bz2",
      "run_exports": {
        "weak": [
          "giflib >=5.1.4,<5.2.0a0"
        ]
      },
      "source_url": "http://downloads.sourceforge.net/project/giflib/giflib-5.1.4.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "Library for reading and writing gif images",
      "text_prefix": true,
      "version": "5.1.4"
    },
    "libblitz": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "https://sourceforge.net/projects/blitz",
      "license": "LGPL 3",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/libblitz-0.10-3.tar.bz2",
      "run_exports": {},
      "source_url": "http://downloads.sourceforge.net/project/blitz/blitz/Blitz%2B%2B%200.10/blitz-0.10.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "A C++ class library for scientific computing which provides performance on par with Fortran 77/90",
      "text_prefix": true,
      "version": "0.10"
    },
    "libogg": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "description": "OGG media container. This is the native format and the recommended container for Xiph codecs.",
      "doc_url": "https://wiki.xiph.org/",
      "home": "https://www.xiph.org/",
      "license": "BSD3",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/libogg-1.3.2-0.tar.bz2",
      "run_exports": {},
      "source_url": "http://downloads.xiph.org/releases/ogg/libogg-1.3.2.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "OGG media container",
      "text_prefix": true,
      "version": "1.3.2"
    },
    "libsvm": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "https://github.com/cjlin1/libsvm",
      "license": "BSD 3-Clause",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/libsvm-3.21-3.tar.bz2",
      "run_exports": {},
      "source_url": "https://github.com/cjlin1/libsvm/archive/v321.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "A simple, easy-to-use, and efficient software for SVM classification and regression.",
      "text_prefix": false,
      "version": "3.21"
    },
    "openfst": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "description": "OpenFST library was developed by contributors <http://www.openfst.org/twiki/bin/view/FST/FstContributors> from Google Research and NYU's Courant Institute. It is intended to be comprehensive, flexible, efficient and scale well to large problems.",
      "dev_url": "http://www.openfst.org/twiki/bin/view/Forum/FstForum",
      "doc_url": "http://www.openfst.org/doxygen/fst/html",
      "home": "http://www.openfst.org/",
      "license": "Apache-2.0",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "linux-64/openfst-1.6.1-0.tar.bz2",
      "run_exports": {},
      "subdirs": [
        "linux-64"
      ],
      "summary": "A library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs)",
      "text_prefix": false,
      "version": "1.6.1"
    },
    "speex": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "http://www.speex.org",
      "license": "BSD-3-Clause",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/speex-1.2.0-0.tar.bz2",
      "run_exports": {},
      "source_url": "http://downloads.xiph.org/releases/speex/speex-1.2.0.tar.gz",
      "subdirs": [
        "osx-64"
      ],
      "summary": "Speex: A Free Codec For Free Speech",
      "text_prefix": true,
      "version": "1.2.0"
    },
    "toolchain": {
      "activate.d": true,
      "binary_prefix": false,
      "deactivate.d": true,
      "home": "https://github.com/conda-forge/toolchain-feedstock",
      "license": "BSD 3-Clause",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/toolchain-2.4.0-0.tar.bz2",
      "run_exports": {},
      "subdirs": [
        "linux-64",
        "osx-64"
      ],
      "summary": "A meta-package to enable the right toolchain.",
      "text_prefix": false,
      "version": "2.4.0"
    },
    "vlfeat": {
      "activate.d": false,
      "binary_prefix": true,
      "deactivate.d": false,
      "home": "http://www.vlfeat.org/",
      "license": "BSD 2-Clause",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/vlfeat-0.9.21-h2d5a098_0.tar.bz2",
      "run_exports": {
        "weak": [
          "vlfeat >=0.9.21,<0.9.22.0a0"
        ]
      },
      "source_url": "http://www.vlfeat.org/download/vlfeat-0.9.21-bin.tar.gz",
      "subdirs": [
        "linux-64",
        "osx-64"
      ],
      "summary": "The VLFeat open source library implements popular computer vision algorithms specializing in image understanding and local features extraction and matching.",
      "text_prefix": false,
      "version": "0.9.21"
    },
    "x264": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "http://www.videolan.org/developers/x264.html",
      "license": "GPL 2",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/x264-20131217-3.tar.bz2",
      "run_exports": {},
      "source_url": "http://download.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-20131217-2245-stable.tar.bz2",
      "subdirs": [
        "linux-64",
        "osx-64"
      ],
      "summary": "A free software library for encoding video streams into the H.264/MPEG-4 AVC format.",
      "text_prefix": true,
      "version": "20131217"
    },
    "yasm": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "home": "http://www.tortall.net/projects/yasm/",
      "license": "BSD 3-Clause",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "reference_package": "osx-64/yasm-1.3.0-3.tar.bz2",
      "run_exports": {},
      "source_url": "http://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz",
      "subdirs": [
        "linux-64",
        "osx-64"
      ],
      "summary": "Yasm is a complete rewrite of the NASM assembler under the \"new\" BSD License.",
      "text_prefix": false,
      "version": "1.3.0"
    }
  },
  "subdirs": [
    "linux-64",
    "noarch",
    "osx-64"
  ]
}

If I browse to http://beatubulatest.lab.idiap.ch/software/bob/conda/, then I see the list above!

So, conda indexing did something weird there.

@amohammadi: could you please check and cleanup? Thx.

Furthermore, the indexing command is dying:

# systemctl status conda-channel-watch.service
● conda-channel-watch.service - Conda Channel Update
   Loaded: loaded (/etc/systemd/system/conda-channel-watch.service; static; vendo
   Active: inactive (dead) since Tue 2019-07-09 10:34:03 CEST; 10s ago

It's not dying. It's dead :D that's how it is supposed to be.

The channeldata.json is not being updated... Who generates that?

Maybe the new version of conda. I am going to delete it since I downgraded the conda version.

Ah - I see what is happening.

That new magic is generating an index.html file. That file is picked up by the server and is served instead of the real directory listing.

Let me see if there is a way to ignore it.

There - it looks better now. The DirectoryIndexing disabled was in the wrong spot in our configuration.

Now you can see the file and directory listings.

Notice, however, that the file channeldata.json, index.html and rss.xml are still being generated by whatever you are doing and they are just wrong.

The conda index command seems to have changed. I am looking at that.

WARNING: The following dependencies do not exist in the channel
    and are not declared as external dependencies:
  _tflow_180_select
    - linux-64/bob-5.0.0-py27hc3b47e9_0.tar.bz2
    - linux-64/bob-5.0.0-py36hc3b47e9_0.tar.bz2
    - linux-64/bob-devel-2018.07.06-py27hf484d3e_0.tar.bz2
    - linux-64/bob-devel-2018.07.06-py36hf484d3e_0.tar.bz2
    - linux-64/bob-devel-2018.08.22-py27hf484d3e_0.tar.bz2
    - linux-64/bob-devel-2018.08.22-py36hf484d3e_0.tar.bz2
    - linux-64/bob-devel-2018.09.20-py27hf484d3e_0.tar.bz2
    - linux-64/bob-devel-2018.09.20-py36hf484d3e_0.tar.bz2
  _tflow_190_select
...

Looks like I need to use an updated conda version to index the channel. @andre.anjos could you please ask for extra storage for the server? We will probably need as much as space that we have as tarballs on the server ...

@amohammadi: please go ahead and open a ticket with me in CC.

@andre.anjos bob's website is broken now https://www.idiap.ch/software/bob/

This is now fixed.

If you want to access the real directory listings though, you should use:

http://<private-internal-server>/public/conda

Going through the https://www.idiap.ch/software/bob/conda will show you the contents of index.html as generated by the conda index command.

I have downgraded our conda that indexes our channels as a temporary measure.

closed

mentioned in issue #32 (closed)