saliency.rst

Saliency

A saliency map highlights areas of interest within an image. In the context of TB detection, this would be the locations in a chest X-ray image where tuberculosis is present.

This package provides scripts that can generate saliency maps and compute relevant metrics for interpretability purposes.

Some of the scripts require the use of a database with human-annotated saliency information.

Generation

Saliency maps can be generated with the :ref:`saliency generate command <mednet.cli>`. They are represented as numpy arrays of the same size as thes images, with values in the range [0-1] and saved in .npy files.

Several mapping algorithms are available to choose from, which can be specified with the -s option.

Examples

Generates saliency maps for all prediction dataloaders on a DataModule, using a pre-trained pasa model, and saves them as numpy-pickeled objects on the output directory:

mednet classify saliency generate -vv pasa tbx11k-v1-healthy-vs-atb --weight=path/to/model-at-lowest-validation-loss.ckpt --output-folder=path/to/output

Viewing

To overlay saliency maps over the original images, use the :ref:`classify saliency view command <mednet.cli>`. Results are saved as PNG images in which brigter pixels correspond to areas with higher saliency.

Examples

Generates visualizations in form of heatmaps from existing saliency maps for a dataset configuration:

# input-folder is the location of the saliency maps created with `mednet generate`
mednet classify saliency view -vv pasa tbx11k-v1-healthy-vs-atb --input-folder=parent_folder/gradcam/ --output-folder=path/to/visualizations

Interpretability

Given a target label, the interpretability step computes the proportional energy and average saliency focus in a DataModule.

The proportional energy is defined as the quantity of activation that lies within the ground truth boxes compared to the total sum of the activations. The average saliency focus is the sum of the values of the saliency map over the ground-truth bounding boxes, normalized by the total area covered by all ground-truth bounding boxes.

Note

Currently, this functionality requires a DataModule containing human-annotated bounding boxes.

Examples

Evaluate the generated saliency maps for their localization performance:

mednet saliency interpretability -vv tbx11k-v1-healthy-vs-atb --input-folder=parent-folder/saliencies/ --output-json=path/to/interpretability-scores.json

Completeness

The saliency completeness script computes ROAD scores of saliency maps and saves them in a .json file.

The ROAD algorithm ([ROAD-2022]_) estimates the explainability (in the completeness sense) of saliency maps by substituting relevant pixels in the input image by a local average, re-running prediction on the altered image, and measuring changes in the output classification score when said perturbations are in place. By substituting most or least relevant pixels with surrounding averages, the ROAD algorithm estimates the importance of such elements in the produced saliency map.

This requires a DataModule containing human-annotated bounding boxes.

Examples

Calculates the ROAD scores for an existing dataset configuration and stores them in .json files:

mednet classify saliency completeness -vv pasa tbx11k-v1-healthy-vs-atb --device="cuda:0" --weight=path/to/model-at-lowest-validation-loss.ckpt --output-json=path/to/completeness-scores.json

Evaluation

Saliency evaluation step generates tables and plots from the results of the interpretability and completeness steps.

Examples

Tabulates and generates plots for two saliency map algorithms:

mednet saliency evaluate -vv -e gradcam path/to/gradcam-completeness.json path/to/gradcam-interpretability.json -e gradcam++ path/to/gradcam++-completeness.json path/to/gradcam++-interpretability.json