evaluation.rst

Evaluation

To evaluate trained models use use bob binseg test followed by the model config, the dataset config and the path to the pretrained model via the argument -w.

Alternatively point to the output folder used during training via the -o argument. The Checkpointer will load the model as indicated in the file: last_checkpoint.

Use bob binseg test --help for more information.

E.g. run inference on model M2U-Net on the DRIVE test set:

# Point directly to saved model via -w argument:
bob binseg test M2UNet DRIVETEST -o /outputfolder/for/results -w /direct/path/to/weight/model_final.pth

# Use training output path (requries last_checkpoint file to be present)
# The evaluation results will be stored in the same folder
bob binseg test M2UNet DRIVETEST -o /DRIVE/M2UNet/output

Outputs

The inference run generates the following output files:

.
├── images  # the predicted probabilities as grayscale images in .png format
├── hdf5    # the predicted probabilties in hdf5 format
├── last_checkpoint  # text file that keeps track of the last checkpoint
├── M2UNet_trainlog.csv # training log
├── M2UNet_trainlog.pdf # training log plot
├── model_*.pth # model checkpoints
└── results
    ├── image*.jpg.csv # evaluation metrics for each image
    ├── Metrics.csv # average evaluation metrics
    ├── ModelSummary.txt # model summary and parameter count
    ├── precision_recall.pdf # precision vs recall plot
    └── Times.txt # inference times

Inference Only Mode

If you wish to run inference only on a folder containing images, use the predict function in combination with a :ref:`bob.ip.binseg.configs.datasets.imagefolderinference` config. E.g.:

bob binseg predict M2UNet /path/to/myinferencedatasetconfig.py -b 1 -d cpu -o /my/output/path -w /path/to/pretrained/weight/model_final.pth -vv

Pretrained Models

Due to storage limitations we only provide weights of a subset of all evaluated models:

	DRIU	M2UNet
DRIVE	`DRIU_DRIVE.pth`_	`M2UNet_DRIVE.pth <m2unet_drive.pth_>`_ <m2unet_drive.pth_>
COVD-DRIVE		M2UNet_COVD-DRIVE.pth
COVD-DRIVE SSL		M2UNet_COVD-DRIVE_SSL.pth
STARE	DRIU_STARE.pth_	M2UNet_STARE.pth
COVD-STARE		M2UNet_COVD-STARE.pth
COVD-STARE SSL		M2UNet_COVD-STARE_SSL.pth
CHASE_DB1	DRIU_CHASEDB1.pth_	M2UNet_CHASEDB1.pth
COVD-CHASE_DB1		M2UNet_COVD-CHASEDB1.pth
COVD-CHASE_DB1 SSL		M2UNet_COVD-CHASEDB1_SSL.pth
IOSTARVESSEL	DRIU_IOSTAR.pth_	M2UNet_IOSTARVESSEL.pth
COVD-IOSTAR		M2UNet_COVD-IOSTAR.pth
COVD-IOSTAR SSL		M2UNet_COVD-IOSTAR_SSL.pth
HRF	DRIU_HRF.pth_	M2UNet_HRF1168.pth
COVD-HRF		M2UNet_COVD-HRF.pth
COVD-HRF SSL		M2UNet_COVD-HRF_SSL.pth

To run evaluation of pretrained models pass url as -w argument. E.g.:

bob binseg test DRIU DRIVETEST -o Evaluation_DRIU_DRIVE -w https://www.idiap.ch/software/bob/data/bob/bob.ip.binseg/master/DRIU_DRIVE.pth
bob binseg test M2UNet DRIVETEST -o Evaluation_M2UNet_DRIVE -w https://www.idiap.ch/software/bob/data/bob/bob.ip.binseg/master/M2UNet_DRIVE.pth