Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images Based on Physiological and Anatomical Cues

Verena Jasmin Hallitschke, Tobias Schlumberger, Philipp Kataliakos, Zdravko Marinov, Moon Kim, Lars Heiliger, Constantin Seibold, Jens Kleesiek, Rainer Stiefelhagen

Introduction

Deep learning models for medical imaging require vast amounts of voxel-wise annotations, a task that is particularly labor-intensive for 3D volumetric data like PET/CT scans. Lung lesions often present a unique challenge: they may appear as clear metabolic "hot spots" in PET but have ambiguous boundaries in CT, or conversely, CT may show anatomical structures that PET lacks the resolution to define. This paper introduces a multimodal interactive framework designed to leverage both physiological and anatomical cues to streamline the annotation process.

Methodology

The proposed framework builds upon the DeepIGeoS architecture, utilizing a two-stage approach. First, a Proposal Network (P-Net) generates an initial automated segmentation from concatenated PET and CT volumes. Second, a Refinement Network (R-Net) allows users to iteratively improve the mask through foreground and background "scribbles."

Overview of the framework components including the multimodal interface, training simulation, and RoI-based geodesic transformation.

Technical Innovations

To optimize the interactive loop, the authors introduced two key innovations:

  1. Ellipsoid-based User Simulation: To train the refinement network without constant human input, the system simulates expert clicks using random ellipsoids. Foreground ellipsoids are placed within ground truth boundaries, while background ellipsoids target over-segmented areas to mimic realistic error correction.
  2. Region-of-Interest (RoI) Geodesic Distance Transform (GDT): Traditional GDT calculations across entire 3D volumes are computationally prohibitive for real-time interaction. By restricting calculations to a local RoI around user clicks, the authors achieved a 79x speedup (from ~98s to 1.25s) with negligible impact on accuracy.

The authors utilized a subset of 152 studies from the AutoPET dataset, featuring whole-body PET/CT scans, for training and internal validation.

Multimodal Interface and User Study

The framework was implemented as a 3D Slicer plugin featuring a "Two-View" interface. This side-by-side linked view allows the mouse cursor to appear simultaneously in both PET and CT slices, reducing the cognitive load required to cross-reference metabolic activity with anatomical boundaries.

The side-by-side linked 'Two-View' interface showing PET and CT scans simultaneously.

A user study involving four medical experts compared this multimodal approach against the standard GraphCut method. The results indicated that the "Two-View" interface consistently yielded higher Dice scores and faster annotation times.

Results and Generalization

The framework demonstrated significant efficiency gains, saving approximately one minute per volume compared to traditional methods—a 12.5% increase in throughput. Furthermore, the model showed robust generalization when tested on the external Lung-PET-CT-Dx dataset, successfully refining automated proposals on unseen data.

Conclusion

By combining the physiological insights of PET with the anatomical precision of CT in a high-speed interactive loop, this framework addresses the bottleneck of 3D medical image annotation. Future work will explore uncertainty-guided suggestions to further direct user attention to the most critical areas for refinement.

Summary updated

BibTeX
@inproceedings{hallitschke2023multimodal,
  author = {Verena Jasmin Hallitschke and Tobias Schlumberger and Philipp Kataliakos and Zdravko Marinov and Moon Kim and Lars Heiliger and Constantin Seibold and Jens Kleesiek and Rainer Stiefelhagen},
  title = {{Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images Based on Physiological and Anatomical Cues}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230334},
}
Figure