Introduction
Medical image segmentation—the process of delineating specific tissues or organs—is a cornerstone of modern diagnostic and surgical planning. However, it remains a significant bottleneck in clinical workflows. Traditional deep learning approaches typically require massive, manually labeled datasets for every specific anatomical structure and imaging modality. While the original Segment Anything Model (SAM1) introduced a powerful "point-and-click" interface for 2D segmentation, it lacked the spatial continuity required for 3D volumes. In a typical MRI scan consisting of hundreds of slices, SAM1 requires prompts for every individual slice, a task that remains prohibitively time-consuming.
This research explores a novel adaptation of Meta’s Segment Anything Model 2 (SAM2). By treating the anatomical slices of a 3D MRI as consecutive frames in a video, the authors leverage SAM2’s temporal memory mechanism to achieve high-accuracy volumetric segmentation with minimal human intervention.
Methodology: Spatial Depth as Time
The fundamental insight of this work is that the "temporal" dimension in video is mathematically analogous to the "depth" dimension in 3D medical imaging. SAM2’s architecture, designed to track objects across video frames, can be repurposed to "track" anatomy across the depth of a 3D scan.
The proposed workflow consists of two primary stages:
- Initial Prompting: A user provides a sparse prompt (e.g., a single point or bounding box) on a representative "middle" slice of the MRI volume.
- Propagation: Utilizing SAM2’s streaming memory mechanism and Hiera transformer architecture, the model segments the initial slice and propagates the resulting mask forward and backward through the entire 3D volume.
Experimental Setup
The study utilized the OAI-ZIB dataset, comprising 488 knee MRI volumes from the Osteoarthritis Initiative. These high-resolution scans (matrix size $160 \times 384 \times 384$) were acquired via 3D double-echo steady state (DESS) sequences. Ground truth masks for the femur and tibia, provided by the Zuse Institute Berlin, served as the benchmark for evaluation. The researchers compared various configurations, including model sizes (tiny to large), prompt schemes (points vs. bounding boxes), and the performance of SAM1 versus SAM2.
Results and Discussion
The experimental results demonstrate that SAM2’s video-tracking capabilities significantly enhance segmentation efficiency and accuracy compared to static image models.
Quantitative Performance
The "3 points + video" prompting scheme achieved a median Dice Similarity Coefficient (DSC) of 0.9196 for the combined femur and tibia segmentation. As shown in the performance table, activating the video predictor allows SAM2 to far outstrip SAM1’s capabilities. For instance, providing just three points on a single slice and allowing SAM2 to propagate the results proved more effective than providing hundreds of manual points for SAM1 across multiple slices.
Key Insights
- Efficiency: The model's internal memory of anatomical shapes is remarkably robust, often outperforming sparse human prompts.
- Model Scaling: Interestingly, the
sam2_hiera_base_plusmodel frequently matched or exceeded the performance of the much largerlargevariant. This suggests that for specific medical tasks, architectural efficiency may be more important than raw parameter count. - Prompt Robustness: While single-point prompts occasionally led to "foreground leakage" (selecting the entire leg), the use of negative points or bounding boxes effectively constrained the model to the target anatomy.
Conclusion
This study demonstrates that SAM2 can be effectively adapted for zero-shot 3D medical image segmentation by reframing spatial depth as temporal progression. This approach significantly reduces the manual labor required for volumetric analysis while maintaining high accuracy. Future research will focus on fine-tuning SAM2 on medical-specific datasets and expanding the workflow to other modalities such as CT and ultrasound to further validate its generalizability in clinical settings.
Summary updated