From c95a0e994c71bc26b3cd069c1b32598eff386f42 Mon Sep 17 00:00:00 2001
From: Andre Anjos <andre.dos.anjos@gmail.com>
Date: Sat, 21 Mar 2020 21:41:22 +0100
Subject: [PATCH] [doc] Better tables with list-table directive; Fix all
warnings
---
doc/acknowledgements.rst | 40 ++-----
doc/api.rst | 2 +
doc/benchmarkresults.rst | 71 ++++++++----
doc/configs.rst | 235 ---------------------------------------
doc/covdresults.rst | 97 ++++++++++------
doc/datasets.rst | 164 ++++++++++++++++++++++-----
doc/evaluation.rst | 10 +-
doc/index.rst | 1 -
doc/links.rst | 1 +
doc/references.rst | 14 +++
10 files changed, 283 insertions(+), 352 deletions(-)
delete mode 100644 doc/configs.rst
diff --git a/doc/acknowledgements.rst b/doc/acknowledgements.rst
index d273b8bf..9783907f 100644
--- a/doc/acknowledgements.rst
+++ b/doc/acknowledgements.rst
@@ -1,39 +1,17 @@
.. -*- coding: utf-8 -*-
+
.. _bob.ip.binseg.acknowledgements:
-================
-Acknowledgements
-================
+==================
+ Acknowledgements
+==================
This packages utilizes code from the following packages:
-* The model-checkpointer is based on the Checkpointer in maskrcnn_benchmark by::
-
- @misc{massa2018mrcnn,
- author = {Massa, Francisco and Girshick, Ross},
- title = {{maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch}},
- year = {2018},
- howpublished = {\url{https://github.com/facebookresearch/maskrcnn-benchmark}},
- note = {Accessed: 2019.05.01}
- }
-
-* The AdaBound optimizer code by::
-
- @inproceedings{Luo2019AdaBound,
- author = {Luo, Liangchen and Xiong, Yuanhao and Liu, Yan and Sun, Xu},
- title = {Adaptive Gradient Methods with Dynamic Bound of Learning Rate},
- booktitle = {Proceedings of the 7th International Conference on Learning Representations},
- month = {May},
- year = {2019},
- address = {New Orleans, Louisiana}
- }
+* The model-checkpointer is based on the implementation in
+ `maskrcnn-benchmark`_ by [MASSA-2018]_
+* The AdaBound optimizer code was sourced from [LUO-2019]_
+* The MobileNetV2 backbone is based on [LIN-2018]_
-* The MobileNetV2 backbone is based on an implementation by::
- @misc{tonylins,
- author = {Ji Lin},
- title = {pytorch-mobilenet-v2},
- year = {2018}
- howpublished = {\url{https://github.com/tonylins/pytorch-mobilenet-v2}},
- note = {Accessed: 2019.05.01}
- }
+.. include:: links.rst
diff --git a/doc/api.rst b/doc/api.rst
index a79deca5..5e836692 100644
--- a/doc/api.rst
+++ b/doc/api.rst
@@ -116,6 +116,8 @@ Models
bob.ip.binseg.configs.models.unet
+.. _bob.ip.binseg.configs.datasets:
+
Datasets
========
diff --git a/doc/benchmarkresults.rst b/doc/benchmarkresults.rst
index 2f391611..becbf3b3 100644
--- a/doc/benchmarkresults.rst
+++ b/doc/benchmarkresults.rst
@@ -1,30 +1,59 @@
.. -*- coding: utf-8 -*-
-.. _bob.ip.binseg.benchmarkresults:
+.. _bob.ip.binseg.benchmarkresults:
-==================
-Benchmark Results
-==================
+===================
+ Benchmark Results
+===================
-F1 Scores
-===========
+F1 Scores (micro-level)
+-----------------------
-* Benchmark results for models: DRIU, HED, M2UNet and U-Net.
-* Models are trained and tested on the same dataset using the train-test split as indicated in :ref:`bob.ip.binseg.datasets`
+* Benchmark results for models: DRIU, HED, M2U-Net and U-Net.
+* Models are trained and tested on the same dataset using the
+ train-test split as indicated in :ref:`bob.ip.binseg.configs.datasets` (i.e.,
+ these are *intra*-datasets tests)
* Standard-deviations across all test images are indicated in brakets
-
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-| F1 (std) | :ref:`bob.ip.binseg.configs.datasets.chasedb1` | :ref:`bob.ip.binseg.configs.datasets.drive` | :ref:`bob.ip.binseg.configs.datasets.hrf` | :ref:`bob.ip.binseg.configs.datasets.iostar` | :ref:`bob.ip.binseg.configs.datasets.stare` |
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-| :ref:`bob.ip.binseg.configs.models.driu` | `0.810 (0.021) <driu_chasedb1.pth_>`_ | `0.820 (0.014) <driu_drive.pth_>`_ | `0.783 (0.055) <driu_hrf.pth_>`_ | `0.825 (0.020) <driu_iostar.pth_>`_ | `0.827 (0.037) <driu_stare.pth_>`_ |
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-| :ref:`bob.ip.binseg.configs.models.hed` | 0.810 (0.022) | 0.817 (0.013) | 0.783 (0.058) | 0.825 (0.020) | 0.823 (0.037) |
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-| :ref:`bob.ip.binseg.configs.models.m2unet` | `0.802 (0.019) <m2unet_chasedb1.pth_>`_ | `0.803 (0.014) <m2unet_drive.pth_>`_ | `0.780 (0.057) <m2unet_hrf.pth_>`_ | `0.817 (0.020) <m2unet_iostar.pth_>`_ | `0.815 (0.041) <m2unet_stare.pth_>`_ |
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-| :ref:`bob.ip.binseg.configs.models.unet` | 0.812 (0.020) | 0.822 (0.015) | 0.788 (0.051) | 0.818 (0.019) | 0.829 (0.042) |
-+--------------------------------------------+------------------------------------------------+---------------------------------------------+-------------------------------------------+----------------------------------------------+---------------------------------------------+
-
+* Database and Model links (table top row and left column) are linked to the
+ originating configuration files used to obtain these results.
+* For some results, the actual deep neural network models are provided (by
+ clicking on the associated F1 Score).
+* Check `our paper`_ for details on the calculation of the F1 Score and standard
+ deviations.
+
+.. list-table::
+ :header-rows: 1
+
+ * - F1 (std)
+ - :py:mod:`DRIU <bob.ip.binseg.configs.models.driu>`
+ - :py:mod:`HED <bob.ip.binseg.configs.models.hed>`
+ - :py:mod:`M2U-Net <bob.ip.binseg.configs.models.m2unet>`
+ - :py:mod:`U-Net <bob.ip.binseg.configs.models.unet>`
+ * - :py:mod:`CHASE-DB1 <bob.ip.binseg.configs.datasets.chasedb1>`
+ - `0.810 (0.021) <driu_chasedb1.pth_>`_
+ - 0.810 (0.022)
+ - `0.802 (0.019) <m2unet_chasedb1.pth_>`_
+ - 0.812 (0.020)
+ * - :py:mod:`DRIVE <bob.ip.binseg.configs.datasets.drive>`
+ - `0.820 (0.014) <driu_drive.pth_>`_
+ - 0.817 (0.013)
+ - `0.803 (0.014) <m2unet_drive.pth_>`_
+ - 0.822 (0.015)
+ * - :py:mod:`HRF <bob.ip.binseg.configs.datasets.hrf1168>`
+ - `0.783 (0.055) <driu_hrf.pth_>`_
+ - 0.783 (0.058)
+ - `0.780 (0.057) <m2unet_hrf.pth_>`_
+ - 0.788 (0.051)
+ * - :py:mod:`IOSTAR (vessel) <bob.ip.binseg.configs.datasets.iostarvessel>`
+ - `0.825 (0.020) <driu_iostar.pth_>`_
+ - 0.825 (0.020)
+ - `0.817 (0.020) <m2unet_iostar.pth_>`_
+ - 0.818 (0.019)
+ * - :py:mod:`STARE <bob.ip.binseg.configs.datasets.stare>`
+ - `0.827 (0.037) <driu_stare.pth_>`_
+ - 0.823 (0.037)
+ - `0.815 (0.041) <m2unet_stare.pth_>`_
+ - 0.829 (0.042)
.. include:: links.rst
diff --git a/doc/configs.rst b/doc/configs.rst
deleted file mode 100644
index 93e09612..00000000
--- a/doc/configs.rst
+++ /dev/null
@@ -1,235 +0,0 @@
-.. -*- coding: utf-8 -*-
-.. _bob.ip.binseg.configs:
-
-===============
-Configs
-===============
-
-Dataset Configs
-===============
-
-We provide variants for the training and test sets of each supported database,
-as well as versions for COVD- (COmbined training sets of all publicly available
-Vessel Dataset without target dataset) and SSL (Semi-supervised Learning), as
-explained in our report.
-
-.. _bob.ip.binseg.configs.datasets.imagefolder:
-
-ImageFolder
------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/imagefolder.py
-
-
-.. _bob.ip.binseg.configs.datasets.imagefoldertest:
-
-ImageFolderTest
----------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/imagefoldertest.py
-
-
-.. _bob.ip.binseg.configs.datasets.imagefolderinference:
-
-ImageFolderInference
---------------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/imagefolderinference.py
-
-
-.. _bob.ip.binseg.configs.datasets.chasedb1:
-
-CHASEDB1
---------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/chasedb1.py
-
-
-.. _bob.ip.binseg.configs.datasets.chasedb1test:
-
-CHASEDB1TEST
-------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/chasedb1test.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-drive:
-
-COVD-DRIVE
-----------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/starechasedb1iostarhrf544.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-drive_ssl:
-
-COVD-DRIVE_SSL
---------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/starechasedb1iostarhrf544ssldrive.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-stare:
-
-COVD-STARE
-----------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivechasedb1iostarhrf608.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-stare_ssl:
-
-COVD-STARE_SSL
---------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivechasedb1iostarhrf608sslstare.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-iostar:
-
-COVD-IOSTARVESSEL
------------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestarechasedb1hrf1024.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-iostar_ssl:
-
-COVD-IOSTARVESSEL_SSL
----------------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestarechasedb1hrf1024ssliostar.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-hrf:
-
-COVD-HRF
---------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestarechasedb1iostar1168.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-hrf_ssl:
-
-COVD-HRF_SSL
-------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestarechasedb1iostar1168sslhrf.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-chasedb1:
-
-COVD-CHASEDB1
--------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestareiostarhrf960.py
-
-
-.. _bob.ip.binseg.configs.datasets.covd-chasedb1_ssl:
-
-COVD-CHASEDB1_SSL
------------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivestareiostarhrf960.py
-
-
-.. _bob.ip.binseg.configs.datasets.drive:
-
-DRIVE
------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drive.py
-
-
-.. _bob.ip.binseg.configs.datasets.drivetest:
-
-DRIVETEST
----------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/drivetest.py
-
-
-.. _bob.ip.binseg.configs.datasets.hrf:
-
-HRF
----
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/hrf1168.py
-
-
-.. _bob.ip.binseg.configs.datasets.hrftest:
-
-HRFTEST
--------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/hrftest.py
-
-
-.. _bob.ip.binseg.configs.datasets.iostar:
-
-IOSTARVESSEL
-------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/iostarvessel.py
-
-
-.. _bob.ip.binseg.configs.datasets.iostarvesseltest:
-
-IOSTARVESSELTEST
-----------------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/iostarvesseltest.py
-
-
-.. _bob.ip.binseg.configs.datasets.stare:
-
-STARE
------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/stare.py
-
-
-.. _bob.ip.binseg.configs.datasets.staretest:
-
-STARETEST
----------
-.. literalinclude:: ../bob/ip/binseg/configs/datasets/staretest.py
-
-
-
-Model Configs
-==============
-
-
-.. _bob.ip.binseg.configs.models.driu:
-
-DRIU
-----
-.. literalinclude:: ../bob/ip/binseg/configs/models/driu.py
-
-
-.. _bob.ip.binseg.configs.models.driubn:
-
-DRIUBN
-------
-.. literalinclude:: ../bob/ip/binseg/configs/models/driubn.py
-
-
-.. _bob.ip.binseg.configs.models.hed:
-
-HED
----
-.. literalinclude:: ../bob/ip/binseg/configs/models/hed.py
-
-
-.. _bob.ip.binseg.configs.models.m2unet:
-
-M2UNet
-------
-.. literalinclude:: ../bob/ip/binseg/configs/models/m2unet.py
-
-
-.. _bob.ip.binseg.configs.models.unet:
-
-UNet
-----
-.. literalinclude:: ../bob/ip/binseg/configs/models/unet.py
-
-
-.. _bob.ip.binseg.configs.models.driussl:
-
-DRIUSSL
--------
-.. literalinclude:: ../bob/ip/binseg/configs/models/driussl.py
-
-
-.. _bob.ip.binseg.configs.models.driubnssl:
-
-DRIUBNSSL
----------
-.. literalinclude:: ../bob/ip/binseg/configs/models/driubnssl.py
-
-
-.. _bob.ip.binseg.configs.models.m2unetssl:
-
-M2UNetSSL
----------
-.. literalinclude:: ../bob/ip/binseg/configs/models/m2unetssl.py
diff --git a/doc/covdresults.rst b/doc/covdresults.rst
index 4eb4c120..73aa7270 100644
--- a/doc/covdresults.rst
+++ b/doc/covdresults.rst
@@ -7,44 +7,73 @@
============================
In addition to the M2U-Net architecture, we also evaluated the larger DRIU
-network and a variation of it that contains batch normalization (DRIU BN) on
-COVD- and COVD-SSL. Perhaps surprisingly, for the majority of combinations, the
-performance of the DRIU variants are roughly equal or worse than the M2U-Net.
-We anticipate that one reason for this could be overparameterization of large
-VGG16 models that are pretrained on ImageNet.
+network and a variation of it that contains batch normalization (DRIU+BN) on
+COVD- (Combined Vessel Dataset from all training data minus target test set)
+and COVD-SSL (COVD- and Semi-Supervised Learning). Perhaps surprisingly, for
+the majority of combinations, the performance of the DRIU variants are roughly
+equal or worse to the ones obtained with the much smaller M2U-Net. We
+anticipate that one reason for this could be overparameterization of large
+VGG-16 models that are pretrained on ImageNet.
+
F1 Scores
-=========
-
-Comparison of F1-micro-scores (std) of DRIU and M2U-Net on COVD- and COVD-SSL.
-Standard deviation across test-images in brackets.
-
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| F1 score | :ref:`bob.ip.binseg.configs.models.driu`/:ref:`bob.ip.binseg.configs.models.driussl` | :ref:`bob.ip.binseg.configs.models.driubn`/:ref:`bob.ip.binseg.configs.models.driubnssl` | :ref:`bob.ip.binseg.configs.models.m2unet`/:ref:`bob.ip.binseg.configs.models.m2unetssl` |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-drive` | 0.788 (0.018) | 0.797 (0.019) | `0.789 (0.018) <m2unet_covd-drive.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-drive_ssl` | 0.785 (0.018) | 0.783 (0.019) | `0.791 (0.014) <m2unet_covd-drive_ssl.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-stare` | 0.778 (0.117) | 0.778 (0.122) | `0.812 (0.046) <m2unet_covd-stare.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-stare_ssl` | 0.788 (0.102) | 0.811 (0.074) | `0.820 (0.044) <m2unet_covd-stare_ssl.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-chasedb1` | 0.796 (0.027) | 0.791 (0.025) | `0.788 (0.024) <m2unet_covd-chasedb1.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-chasedb1_ssl` | 0.796 (0.024) | 0.798 (0.025) | `0.799 (0.026) <m2unet_covd-chasedb1_ssl.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-hrf` | 0.799 (0.044) | 0.800 (0.045) | `0.802 (0.045) <m2unet_covd-hrf.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-hrf_ssl` | 0.799 (0.044) | 0.784 (0.048) | `0.797 (0.044) <m2unet_covd-hrf_ssl.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-iostar` | 0.791 (0.021) | 0.777 (0.032) | `0.793 (0.015) <m2unet_covd-iostar.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
-| :ref:`bob.ip.binseg.configs.datasets.covd-iostar_ssl` | 0.797 (0.017) | 0.811 (0.074) | `0.785 (0.018) <m2unet_covd-iostar_ssl.pth>`_ |
-+---------------------------------------------------------+--------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
+---------
+
+Comparison of F1 Scores (micro-level and standard deviation) of DRIU and
+M2U-Net on COVD- and COVD-SSL. Standard deviation across test-images in
+brackets.
+
+.. list-table::
+ :header-rows: 1
+
+ * - F1 score
+ - :py:mod:`DRIU <bob.ip.binseg.configs.models.driu>`/:py:mod:`DRIU@SSL <bob.ip.binseg.configs.models.driussl>`
+ - :py:mod:`DRIU+BN <bob.ip.binseg.configs.models.driubn>`/:py:mod:`DRIU+BN@SSL <bob.ip.binseg.configs.models.driubnssl>`
+ - :py:mod:`M2U-Net <bob.ip.binseg.configs.models.m2unet>`/:py:mod:`M2U-Net@SSL <bob.ip.binseg.configs.models.m2unetssl>`
+ * - :py:mod:`DRIVE (COVD-) <bob.ip.binseg.configs.datasets.starechasedb1iostarhrf544>`
+ - 0.788 (0.018)
+ - 0.797 (0.019)
+ - `0.789 (0.018) <m2unet_covd-drive.pth>`_
+ * - :py:mod:`DRIVE (SSL, COVD-) <bob.ip.binseg.configs.datasets.starechasedb1iostarhrf544ssldrive>`
+ - 0.785 (0.018)
+ - 0.783 (0.019)
+ - `0.791 (0.014) <m2unet_covd-drive_ssl.pth>`_
+ * - :py:mod:`STARE (COVD-) <bob.ip.binseg.configs.datasets.drivechasedb1iostarhrf608>`
+ - 0.778 (0.117)
+ - 0.778 (0.122)
+ - `0.812 (0.046) <m2unet_covd-stare.pth>`_
+ * - :py:mod:`STARE (SSL, COVD-) <bob.ip.binseg.configs.datasets.drivechasedb1iostarhrf608sslstare>`
+ - 0.788 (0.102)
+ - 0.811 (0.074)
+ - `0.820 (0.044) <m2unet_covd-stare_ssl.pth>`_
+ * - :py:mod:`CHASE-DB1 (COVD-) <bob.ip.binseg.configs.datasets.drivestareiostarhrf960>`
+ - 0.796 (0.027)
+ - 0.791 (0.025)
+ - `0.788 (0.024) <m2unet_covd-chasedb1.pth>`_
+ * - :py:mod:`CHASE-DB1 (SSL, COVD-) <bob.ip.binseg.configs.datasets.drivestareiostarhrf960sslchase>`
+ - 0.796 (0.024)
+ - 0.798 (0.025)
+ - `0.799 (0.026) <m2unet_covd-chasedb1_ssl.pth>`_
+ * - :py:mod:`HRF (COVD-) <bob.ip.binseg.configs.datasets.drivestarechasedb1iostar1168>`
+ - 0.799 (0.044)
+ - 0.800 (0.045)
+ - `0.802 (0.045) <m2unet_covd-hrf.pth>`_
+ * - :py:mod:`HRF (SSL, COVD-) <bob.ip.binseg.configs.datasets.drivestarechasedb1iostar1168sslhrf>`
+ - 0.799 (0.044)
+ - 0.784 (0.048)
+ - `0.797 (0.044) <m2unet_covd-hrf_ssl.pth>`_
+ * - :py:mod:`IOSTAR (vessel, COVD-) <bob.ip.binseg.configs.datasets.drivestarechasedb1hrf1024>`
+ - 0.791 (0.021)
+ - 0.777 (0.032)
+ - `0.793 (0.015) <m2unet_covd-iostar.pth>`_
+ * - :py:mod:`IOSTAR (vessel, SSL, COVD-) <bob.ip.binseg.configs.datasets.drivestarechasedb1hrf1024ssliostar>`
+ - 0.797 (0.017)
+ - 0.811 (0.074)
+ - `0.785 (0.018) <m2unet_covd-iostar_ssl.pth>`_
+
M2U-Net Precision vs. Recall Curves
-===================================
+-----------------------------------
Precision vs. recall curves for each evaluated dataset. Note that here the
F1-score is calculated on a macro level (see paper for more details).
diff --git a/doc/datasets.rst b/doc/datasets.rst
index dece6057..bd77ab1f 100644
--- a/doc/datasets.rst
+++ b/doc/datasets.rst
@@ -12,29 +12,141 @@ can be downloaded. We include the reference of the data split protocols used
to generate iterators for training and testing.
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| Dataset | Reference | ``bob.db`` package | H x W | Samples | Mask | Vessel | OD | Cup | Split Reference | Train | Test |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| DRIVE_ | [DRIVE-2004]_ | ``bob.db.drive`` | 584 x 565 | 40 | x | x | | | [DRIVE-2004]_ | 20 | 20 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| STARE_ | [STARE-2000]_ | ``bob.db.stare`` | 605 x 700 | 20 | | x | | | [MANINIS-2016]_ | 10 | 10 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| CHASE-DB1_ | [CHASEDB1-2012]_ | ``bob.db.chasedb`` | 960 x 999 | 28 | | x | | | [CHASEDB1-2012]_ | 8 | 20 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| HRF_ | [HRF-2013]_ | ``bob.db.hrf`` | 2336 x 3504 | 45 | x | x | | | [ORLANDO-2017]_ | 15 | 30 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| IOSTAR_ | [IOSTAR-2016]_ | ``bob.db.iostar`` | 1024 x 1024 | 30 | x | x | x | | [MEYER-2017]_ | 20 | 10 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| DRIONS-DB_ | [DRIONSDB-2008]_ | ``bob.db.drionsdb`` | 400 x 600 | 110 | | | x | | [MANINIS-2016]_ | 60 | 50 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| `RIM-ONE r3`_ | [RIMONER3-2015]_ | ``bob.db.rimoner3`` | 1424 x 1072 | 159 | | | x | x | [MANINIS-2016]_ | 99 | 60 |
-+-----------------+-------------------+------------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| Drishti-GS1_ | [DRISHTIGS1-2014]_ | ``bob.db.drishtigs1`` | varying | 101 | | | x | x | [DRISHTIGS1-2014]_ | 50 | 51 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| REFUGE_ (train) | [REFUGE-2018]_ | ``bob.db.refuge`` | 2056 x 2124 | 400 | | | x | x | [REFUGE-2018]_ | 400 | |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
-| REFUGE_ (val) | [REFUGE-2018]_ | ``bob.db.refuge`` | 1634 x 1634 | 400 | | | x | x | [REFUGE-2018]_ | | 400 |
-+-----------------+--------------------+-----------------------+-------------+---------+------+--------+-----+-----+--------------------+-------+------+
+.. list-table::
+ :header-rows: 1
+
+ * - Dataset
+ - Reference
+ - ``bob.db`` package
+ - H x W
+ - Samples
+ - Mask
+ - Vessel
+ - OD
+ - Cup
+ - Split Reference
+ - Train
+ - Test
+ * - DRIVE_
+ - [DRIVE-2004]_
+ - ``bob.db.drive``
+ - 584 x 565
+ - 40
+ - x
+ - x
+ -
+ -
+ - [DRIVE-2004]_
+ - 20
+ - 20
+ * - STARE_
+ - [STARE-2000]_
+ - ``bob.db.stare``
+ - 605 x 700
+ - 20
+ -
+ - x
+ -
+ -
+ - [MANINIS-2016]_
+ - 10
+ - 10
+ * - CHASE-DB1_
+ - [CHASEDB1-2012]_
+ - ``bob.db.chasedb``
+ - 960 x 999
+ - 28
+ -
+ - x
+ -
+ -
+ - [CHASEDB1-2012]_
+ - 8
+ - 20
+ * - HRF_
+ - [HRF-2013]_
+ - ``bob.db.hrf``
+ - 2336 x 3504
+ - 45
+ - x
+ - x
+ -
+ -
+ - [ORLANDO-2017]_
+ - 15
+ - 30
+ * - IOSTAR_
+ - [IOSTAR-2016]_
+ - ``bob.db.iostar``
+ - 1024 x 1024
+ - 30
+ - x
+ - x
+ - x
+ -
+ - [MEYER-2017]_
+ - 20
+ - 10
+ * - DRIONS-DB_
+ - [DRIONSDB-2008]_
+ - ``bob.db.drionsdb``
+ - 400 x 600
+ - 110
+ -
+ -
+ - x
+ -
+ - [MANINIS-2016]_
+ - 60
+ - 50
+ * - `RIM-ONE r3`_
+ - [RIMONER3-2015]_
+ - ``bob.db.rimoner3``
+ - 1424 x 1072
+ - 159
+ -
+ -
+ - x
+ - x
+ - [MANINIS-2016]_
+ - 99
+ - 60
+ * - Drishti-GS1_
+ - [DRISHTIGS1-2014]_
+ - ``bob.db.drishtigs1``
+ - varying
+ - 101
+ -
+ -
+ - x
+ - x
+ - [DRISHTIGS1-2014]_
+ - 50
+ - 51
+ * - REFUGE_ (train)
+ - [REFUGE-2018]_
+ - ``bob.db.refuge``
+ - 2056 x 2124
+ - 400
+ -
+ -
+ - x
+ - x
+ - [REFUGE-2018]_
+ - 400
+ -
+ * - REFUGE_ (val)
+ - [REFUGE-2018]_
+ - ``bob.db.refuge``
+ - 1634 x 1634
+ - 400
+ -
+ -
+ - x
+ - x
+ - [REFUGE-2018]_
+ -
+ - 400
Folder-based Dataset
@@ -55,7 +167,7 @@ be read via PIL are supported. Additionally, we also support HDF5 binary
files.
For training, a new dataset configuration needs to be created. You can copy the
-template :ref:`bob.ip.binseg.configs.datasets.imagefolder` and amend it
+template :py:mod:`bob.ip.binseg.configs.datasets.imagefolder` and amend it
accordingly, e.g. to point to the the full path of the dataset and if necessary
any preprocessing steps such as resizing, cropping, padding etc.
@@ -66,8 +178,8 @@ Training can then be started with, e.g.:
bob binseg train M2UNet /path/to/myimagefolderconfig.py -b 4 -d cuda -o /my/output/path -vv
Similary for testing, a test dataset config needs to be created. You can copy
-the template :ref:`bob.ip.binseg.configs.datasets.imagefoldertest` and amend it
-accordingly.
+the template :py:mod:`bob.ip.binseg.configs.datasets.imagefoldertest` and amend
+it accordingly.
Testing can then be started with, e.g.:
diff --git a/doc/evaluation.rst b/doc/evaluation.rst
index 2fb923a0..feceb673 100644
--- a/doc/evaluation.rst
+++ b/doc/evaluation.rst
@@ -33,10 +33,10 @@ The inference run generates the following output files:
.. code-block:: bash
.
- ├── images # the predicted probabilities as grayscale images in .png format
+ ├── images # the predicted probabilities as grayscale images in .png format
├── hdf5 # the predicted probabilties in hdf5 format
- ├── last_checkpoint # text file that keeps track of the last checkpoint
- ├── M2UNet_trainlog.csv # training log
+ ├── last_checkpoint # text file that keeps track of the last checkpoint
+ ├── M2UNet_trainlog.csv # training log
├── M2UNet_trainlog.pdf # training log plot
├── model_*.pth # model checkpoints
└── results
@@ -49,7 +49,9 @@ The inference run generates the following output files:
Inference Only Mode
====================
-If you wish to run inference only on a folder containing images, use the ``predict`` function in combination with a :ref:`bob.ip.binseg.configs.datasets.imagefolderinference` config. E.g.:
+If you wish to run inference only on a folder containing images, use the
+``predict`` function in combination with a
+:py:mod:`bob.ip.binseg.configs.datasets.imagefolderinference` config. E.g.:
.. code-block:: bash
diff --git a/doc/index.rst b/doc/index.rst
index 2dfa532d..04c2d851 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -47,7 +47,6 @@ Users Guide
evaluation
benchmarkresults
covdresults
- configs
plotting
visualization
acknowledgements
diff --git a/doc/links.rst b/doc/links.rst
index 5b9a2390..ab294a1e 100644
--- a/doc/links.rst
+++ b/doc/links.rst
@@ -7,6 +7,7 @@
.. _installation: https://www.idiap.ch/software/bob/install
.. _mailing list: https://www.idiap.ch/software/bob/discuss
.. _pytorch: https://pytorch.org
+.. _our paper: https://arxiv.org/abs/1909.03856
.. Raw data websites
.. _drive: https://www.isi.uu.nl/Research/Databases/DRIVE/
diff --git a/doc/references.rst b/doc/references.rst
index d7b4f8d5..97ad5358 100644
--- a/doc/references.rst
+++ b/doc/references.rst
@@ -75,3 +75,17 @@
.. [HE-2015] *S. Xie and Z. Tu*, **Holistically-Nested Edge Detection**, 2015
IEEE International Conference on Computer Vision (ICCV), Santiago, 2015, pp.
1395-1403. https://doi.org/10.1109/ICCV.2015.164
+
+.. [LUO-2019] *L. Luo, Y. Xiong, Y. Liu, and X. Sun*, **Adaptive Gradient
+ Methods with Dynamic Bound of Learning Rate**, Proceedings of the 7th
+ International Conference on Learning Representations (ICLR), Feb. 2019.
+ https://arxiv.org/abs/1902.09843v1
+
+.. [MASSA-2018] *F. Massa and R. Girshick*, **maskrcnn-benchmark: Fast, modular
+ reference implementation of Instance Segmentation and Object Detection
+ algorithms in PyTorch**. 2018. Last accessed: 21.03.2020.
+ https://github.com/facebookresearch/maskrcnn-benchmark
+
+.. [LIN-2018] *J. Lin*, **pytorch-mobilenet-v2: A PyTorch implementation of
+ MobileNetV2**, 2018. Last accessed: 21.03.2020.
+ https://github.com/tonylins/pytorch-mobilenet-v2
--
GitLab