Introduction
Deep learning models for medical imaging require vast amounts of voxel-wise annotations, a task that is particularly labor-intensive for 3D volumetric data like PET/CT scans. Lung lesions often present a unique challenge: they may appear as clear metabolic "hot spots" in PET but have ambiguous boundaries in CT, or conversely, CT may show anatomical structures that PET lacks the resolution to define. This paper introduces a multimodal interactive framework designed to leverage both physiological and anatomical cues to streamline the annotation process.
Methodology
The proposed framework builds upon the DeepIGeoS architecture, utilizing a two-stage approach. First, a Proposal Network (P-Net) generates an initial automated segmentation from concatenated PET and CT volumes. Second, a Refinement Network (R-Net) allows users to iteratively improve the mask through foreground and background "scribbles."
Technical Innovations
To optimize the interactive loop, the authors introduced two key innovations:
- Ellipsoid-based User Simulation: To train the refinement network without constant human input, the system simulates expert clicks using random ellipsoids. Foreground ellipsoids are placed within ground truth boundaries, while background ellipsoids target over-segmented areas to mimic realistic error correction.
- Region-of-Interest (RoI) Geodesic Distance Transform (GDT): Traditional GDT calculations across entire 3D volumes are computationally prohibitive for real-time interaction. By restricting calculations to a local RoI around user clicks, the authors achieved a 79x speedup (from ~98s to 1.25s) with negligible impact on accuracy.
The authors utilized a subset of 152 studies from the AutoPET dataset, featuring whole-body PET/CT scans, for training and internal validation.
Multimodal Interface and User Study
The framework was implemented as a 3D Slicer plugin featuring a "Two-View" interface. This side-by-side linked view allows the mouse cursor to appear simultaneously in both PET and CT slices, reducing the cognitive load required to cross-reference metabolic activity with anatomical boundaries.
A user study involving four medical experts compared this multimodal approach against the standard GraphCut method. The results indicated that the "Two-View" interface consistently yielded higher Dice scores and faster annotation times.
Results and Generalization
The framework demonstrated significant efficiency gains, saving approximately one minute per volume compared to traditional methods—a 12.5% increase in throughput. Furthermore, the model showed robust generalization when tested on the external Lung-PET-CT-Dx dataset, successfully refining automated proposals on unseen data.
Conclusion
By combining the physiological insights of PET with the anatomical precision of CT in a high-speed interactive loop, this framework addresses the bottleneck of 3D medical image annotation. Future work will explore uncertainty-guided suggestions to further direct user attention to the most critical areas for refinement.
Summary updated