Making use of LightningDataModule and simplification of data loading

This merge request adds LightningDataModule to better organize the code and make better use of lightning's features. This centralizes common tasks such as DataLoader creation and application of transforms into a base class to be inherited from.

Data loading was also simplified by removing custom Sample classes and maker functions, and the addition of RuntimeDataset and CachedDataset.

Remaining tasks:

• (@dcarron) Create a common/default DataModule for the shenzhen dataset that takes protocols and transforms as parameters to avoid copying code for each protocol configuration
• (@dcarron) Add typehints to ShenzenDataModule
• (@dcarron) Investigate issue where training a new model with ElasticDeformation as a transform converges more slowly if the data is not cached.
• (@andre.anjos) Update documentation on ShenzenDataModule
• (@biosignal) Update all datasets, using Shenzhen as a reference
  • (@mdelitroz) Montgomery
  • (@mdelitroz) Hivtb
  • (@andre.anjos) Indian
  • (@andre.anjos) Padchest
  • (@mdelitroz) Tbpoc
  • (@andre.anjos) tbx11_simplified -> renamed as tbx11k, protocol v1 (uses original dataset organisation)
  • (@andre.anjos) tbx11_simplified_v2 -> renamed as tbx11k, protocol v2 (uses original dataset organisation)
  • (@andre.anjos) mc_ch -> renamed as montgomery-shenzhen
  • (@andre.anjos) mc_ch_in -> renamed as montgomery-shenzhen-indian
  • (@andre.anjos) mc_ch_in_11k -> renamed as montgomery-shenzhen-indian-tbx11k-v1
  • (@andre.anjos) mc_ch_in_11k_v2 -> renamed as montgomery-shenzhen-indian-tbx11k-v2
  • (@andre.anjos) mc_ch_in_pc -> renamed as montgomery-shenzhen-indian-padchest
  • (@andre.anjos) nih_cxr14_re -> renamed as nih-cxr14 (n.b.: multi-class dataset, radiological findings)
  • (@andre.anjos) nih_cxr14_re_pc -> renamed as nih-cxr14-padchest
• (@dcarron) Update models
• (@dcarron) Update evaluation scripts
• (@dcarron) Update support for extra_validation datasets
• (@biosignal) Update full documentation
• (@biosignal) Update unit tests

Addresses the following issues:

• Closes #14 (closed)
• Closes #18 (closed)
• Closes #19 (closed)
• Closes #21 (closed)
• Closes #29 (closed)

src/ptbench/data/montgomery/__init__.py

-# SPDX-FileCopyrightText: Copyright © 2023 Idiap Research Institute <contact@idiap.ch>
-#
-# SPDX-License-Identifier: GPL-3.0-or-later
-
-"""Montgomery dataset for computer-aided diagnosis.
-
-The Montgomery database has been established to foster research
-in computer-aided diagnosis of pulmonary diseases with a special
-focus on pulmonary tuberculosis (TB).
-
-* Reference: [MONTGOMERY-SHENZHEN-2014]_
-* Original resolution (height x width or width x height): 4020 x 4892
-* Split reference: none
-* Protocol ``default``:
-
-  * Training samples: 64% of TB and healthy CXR (including labels)
-  * Validation samples: 16% of TB and healthy CXR (including labels)
-  * Test samples: 20% of TB and healthy CXR (including labels)
-"""
-
-import importlib.resources
-import os
-
-from ...utils.rc import load_rc
-from .. import make_dataset
-from ..dataset import JSONDataset
-from ..loader import load_pil_baw, make_delayed
-
-_protocols = [
-    importlib.resources.files(__name__).joinpath("default.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_0.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_1.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_2.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_3.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_4.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_5.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_6.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_7.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_8.json.bz2"),
-    importlib.resources.files(__name__).joinpath("fold_9.json.bz2"),
-]
-
-_datadir = load_rc().get("datadir.montgomery", os.path.realpath(os.curdir))
-
-
-def _raw_data_loader(sample):
-    return dict(
-        data=load_pil_baw(os.path.join(_datadir, sample["data"])),  # type: ignore
-        label=sample["label"],
-    )
-
-
-def _loader(context, sample):
-    # "context" is ignored in this case - database is homogeneous
-    # we return delayed samples to avoid loading all images at once
-    return make_delayed(sample, _raw_data_loader)
-
-
-json_dataset = JSONDataset(
-    protocols=_protocols,
-    fieldnames=("data", "label"),
-    loader=_loader,
-)
-"""Montgomery dataset object."""
-
-
-def _maker(protocol, resize_size=512, cc_size=512, RGB=False):
-    from torchvision import transforms
-
-    from ..transforms import ElasticDeformation, RemoveBlackBorders
-
-    post_transforms = []
-    if RGB:
-        post_transforms = [
-            transforms.Lambda(lambda x: x.convert("RGB")),
-            transforms.ToTensor(),
-        ]
-
-    return make_dataset(
-        [json_dataset.subsets(protocol)],
-        [
-            RemoveBlackBorders(),
-            transforms.Resize(resize_size),
-            transforms.CenterCrop(cc_size),
-        ],
-        [ElasticDeformation(p=0.8)],
-        post_transforms,
-    )