Newer
Older
.. SPDX-FileCopyrightText: Copyright © 2023 Idiap Research Institute <contact@idiap.ch>
..
.. SPDX-License-Identifier: GPL-3.0-or-later
.. _ptbench.usage:
=======
Usage
=======
This package supports a fully reproducible research experimentation cycle for
tuberculosis detection with support for the following activities.
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
.. figure:: img/direct_vs_indirect.png
.. _ptbench.usage.direct-detection:
Direct detection
----------------
* Training: Images are fed to a Convolutional Neural Network (CNN),
that is trained to detect the presence of tuberculosis
automatically, via error back propagation. The objective of this phase is to
produce a CNN model.
* Inference (prediction): The CNN is used to generate TB predictions.
* Evaluation: Predications are used to evaluate CNN performance against
provided annotations, and to generate measure files and score tables. Optimal
thresholds are also calculated.
* Comparison: Use predictions results to compare performance of multiple
systems.
.. _ptbench.usage.indirect-detection:
Indirect detection
------------------
* Training (step 1): Images are fed to a Convolutional Neural Network (CNN),
that is trained to detect the presence of radiological signs
automatically, via error back propagation. The objective of this phase is to
produce a CNN model.
* Inference (prediction): The CNN is used to generate radiological signs
predictions.
* Conversion of the radiological signs predictions into a new dataset.
* Training (step 2): Radiological signs are fed to a shallow network, that is
trained to detect the presence of tuberculosis automatically, via error back
propagation. The objective of this phase is to produce a shallow model.
* Inference (prediction): The shallow model is used to generate TB predictions.
* Evaluation: Predications are used to evaluate CNN performance against
provided annotations, and to generate measure files and score tables.
* Comparison: Use predictions results to compare performance of multiple
systems.
We provide :ref:`command-line interfaces (CLI) <ptbench.cli>` that implement
each of the phases above. This interface is configurable using :ref:`exposed's
extensible configuration framework <exposed.config>`. In essence,
each command-line option may be provided as a variable with the same name in a
Python file. Each file may combine any number of variables that are pertinent
to an application.
.. tip::
For reproducibility, we recommend you stick to configuration files when
parameterizing our CLI. Notice some of the options in the CLI interface
(e.g. ``--dataset``) cannot be passed via the actual command-line as it
may require complex Python types that cannot be synthetized in a single
input parameter.
We provide a number of :ref:`preset configuration files <ptbench.config>` that
can be used in one or more of the activities described in this section. Our
command-line framework allows you to refer to these preset configuration files
using special names (a.k.a. "resources"), that procure and load these for you
automatically.
.. _ptbench.usage.commands:
Commands
--------
.. toctree::
:maxdepth: 2
usage/training
usage/evaluation
usage/predtojson
usage/aggregpred