Skip to content

Use pytorch model-zoo style file uploads for `bdt dav upload`

As discussed during the Bob meeting yesterday afternoon, it would be nice to have an option in the upload command for bdt dav that would handle filename hashing automatically, à là pytorch's model zoo implementation (https://pytorch.org/docs/stable/model_zoo.html). This procedure would have to take a file and ensure it is uploaded to the destination URL with a slightly modified name:

if name is file.ext and url is http://example.com/dir/
file is uploaded as http://example.com/dir/file-deadbeef.ext
where `deadbeaf` is the first 8 characters of the original file's sha256 sum.

Similarly, an option in bob/bob.extension's get_data should be able to handle automatic hash checking, if the user specifies the said option, or automatically, based on the filename (e.g. via regular expression matching).

This mechanism provides a way to handle file versions and automatic checking for the completeness of existing cached downloads.