ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images

Marius Schmidt-Mengin, Alexis Benichoux, Shibeshih Mitiku Belachew, Nikos Komodakis, Nikos Paragios

Introduction

3D medical imaging, such as MRI and CT, is a cornerstone of modern diagnostics and treatment planning. However, the manual segmentation of these volumes—identifying and outlining structures like tumors or lesions—is a labor-intensive, expensive, and often subjective task. Weakly supervised learning offers a potential solution by training models on simpler, image-level labels (e.g., "lesion present" vs. "absent") rather than pixel-perfect 3D masks.

The current standard for weak supervision is Class Activation Mapping (CAM). While effective for localization, CAMs are typically derived from deep, downsampled layers of a neural network, resulting in blurry, low-resolution heatmaps that struggle to capture the fine details necessary for clinical applications. This paper introduces ToNNO (Tomographic Reconstruction of a Neural Network’s Output), a framework that leverages the mathematics of tomography to transform 2D classifiers into high-resolution 3D segmenters.

Methodology: Segmentation via Tomography

The core innovation of ToNNO is treating the output of a 2D neural network as a series of projections, analogous to how a CT scanner reconstructs a 3D volume from 1D X-ray projections.

Training with Label Noise

The process begins by training a standard 2D encoder (e.g., ResNet) to classify whether a single slice contains a lesion. Because the available labels are at the volume level, the authors assign the volume's label to every constituent slice. This introduces significant "label noise," as many slices in a "positive" volume will not actually contain the target pathology.

Distribution of logit values for positive and negative volumes.

Inference and Reconstruction

During inference, ToNNO extracts thousands of 2D slices from a 3D volume at various orientations using a Fibonacci lattice. This ensures an even distribution of angles over a sphere. Each slice is processed by the 2D encoder to generate a prediction score (logit).

These scores are then treated as projections. By applying the Inverse Radon Transform—specifically filtered backprojection—the framework reconstructs a 3D heatmap. The "filtering" step, which involves taking the second derivative along the slice axis, is critical. It suppresses the low-frequency blur inherent in simple backprojection and provides the high-frequency detail required for sharp segmentation.

Experimental Results

The researchers evaluated ToNNO across four diverse 3D medical datasets: Multiple Sclerosis (MS) brain MRI, AutoPET-II whole-body PET-CT, MosMedData COVID-19 thoracic CT, and Duke Breast Cancer MRI.

Performance and Resolution

ToNNO consistently outperformed traditional CAM-based methods. On the MS dataset, ToNNO achieved an F1-score of 0.82, significantly higher than the 0.73 achieved by LayerCAM. A hybrid approach, "Tomographic LayerCAM," further improved this to 0.84.

Comparison of segmentation heatmaps: (a) Input MRI, (b) LayerCAM, and (c) ToNNO. Note the significantly higher resolution and localization accuracy of ToNNO.

Crucially, ToNNO is not limited by the resolution of internal feature maps. Because it operates on slice-level predictions, it can generate heatmaps at the original input resolution, making it particularly effective at identifying small, subtle lesions that are often lost in the downsampling layers of standard CAM models.

Discussion and Limitations

One of the most compelling aspects of ToNNO is its mathematical robustness to label noise. While many slices in a positive volume are false positives, their signals do not align across different projection angles. Only true lesions consistently "intersect" during the backprojection process, allowing the actual signal to emerge from the noise.

However, this precision comes at a computational cost. ToNNO requires thousands of inferences per volume, leading to reconstruction times of 40–60 seconds on an RTX 3090, compared to less than a second for slice-wise CAM. Additionally, the authors noted that the choice of normalization in the 2D encoder is vital; Batch Normalization can negatively impact performance by breaking slice independence, whereas Group Normalization or frozen Batch Norm layers are preferred.

Conclusion

ToNNO represents a significant step forward in weakly supervised 3D segmentation. By reframing neural network outputs as tomographic projections, it achieves state-of-the-art performance and high-resolution results without the need for expensive 3D ground-truth masks. Future work may explore iterative reconstruction techniques—common in modern CT imaging—to further refine the quality and efficiency of the reconstructed heatmaps.

Summary updated

Abstract

Annotating lots of 3D medical images for training segmentation models is time-consuming. The goal of weakly supervised semantic segmentation is to train segmentation models without using any ground truth segmentation masks. Our work addresses the case where only image-level categorical labels, indicating the presence or absence of a particular region of interest (such as tumours or lesions), are available. Most existing methods rely on class activation mapping (CAM). We propose a novel approach, ToNNO, which is based on the Tomographic reconstruction of a Neural Network's Output. Our technique extracts stacks of slices with different angles from the input 3D volume, feeds these slices to a 2D encoder, and applies the inverse Radon transform in order to reconstruct a 3D heatmap of the encoder's predictions. This generic method allows to perform dense prediction tasks on 3D volumes using any 2D image encoder. We apply it to weakly supervised medical image segmentation by training the 2D encoder to output high values for slices containing the regions of interest. We test it on four large scale medical image datasets and outperform 2D CAM methods. We then extend ToNNO by combining tomographic reconstruction with CAM methods, proposing Averaged CAM and Tomographic CAM, which obtain even better results.
BibTeX
@inproceedings{schmidtmengin2024tonno,
  author = {Marius Schmidt-Mengin and Alexis Benichoux and Shibeshih Mitiku Belachew and Nikos Komodakis and Nikos Paragios},
  title = {{ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure