Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2

Andrew Seohwan Yu, Mohsen Hariri, Xuecen Zhang, Mingrui Yang, Vipin Chaudhary, Xiaojuan Li

Introduction

Medical image segmentation—the process of delineating specific tissues or organs—is a cornerstone of modern diagnostic and surgical planning. However, it remains a significant bottleneck in clinical workflows. Traditional deep learning approaches typically require massive, manually labeled datasets for every specific anatomical structure and imaging modality. While the original Segment Anything Model (SAM1) introduced a powerful "point-and-click" interface for 2D segmentation, it lacked the spatial continuity required for 3D volumes. In a typical MRI scan consisting of hundreds of slices, SAM1 requires prompts for every individual slice, a task that remains prohibitively time-consuming.

This research explores a novel adaptation of Meta’s Segment Anything Model 2 (SAM2). By treating the anatomical slices of a 3D MRI as consecutive frames in a video, the authors leverage SAM2’s temporal memory mechanism to achieve high-accuracy volumetric segmentation with minimal human intervention.

Methodology: Spatial Depth as Time

The fundamental insight of this work is that the "temporal" dimension in video is mathematically analogous to the "depth" dimension in 3D medical imaging. SAM2’s architecture, designed to track objects across video frames, can be repurposed to "track" anatomy across the depth of a 3D scan.

The proposed workflow consists of two primary stages:

  1. Initial Prompting: A user provides a sparse prompt (e.g., a single point or bounding box) on a representative "middle" slice of the MRI volume.
  2. Propagation: Utilizing SAM2’s streaming memory mechanism and Hiera transformer architecture, the model segments the initial slice and propagates the resulting mask forward and backward through the entire 3D volume.
Figure 1: The proposed workflow adapts SAM2 by treating MRI slices as video frames, allowing a single prompt to propagate into a full volumetric prediction.

Experimental Setup

The study utilized the OAI-ZIB dataset, comprising 488 knee MRI volumes from the Osteoarthritis Initiative. These high-resolution scans (matrix size $160 \times 384 \times 384$) were acquired via 3D double-echo steady state (DESS) sequences. Ground truth masks for the femur and tibia, provided by the Zuse Institute Berlin, served as the benchmark for evaluation. The researchers compared various configurations, including model sizes (tiny to large), prompt schemes (points vs. bounding boxes), and the performance of SAM1 versus SAM2.

Results and Discussion

The experimental results demonstrate that SAM2’s video-tracking capabilities significantly enhance segmentation efficiency and accuracy compared to static image models.

Quantitative Performance

The "3 points + video" prompting scheme achieved a median Dice Similarity Coefficient (DSC) of 0.9196 for the combined femur and tibia segmentation. As shown in the performance table, activating the video predictor allows SAM2 to far outstrip SAM1’s capabilities. For instance, providing just three points on a single slice and allowing SAM2 to propagate the results proved more effective than providing hundreds of manual points for SAM1 across multiple slices.

Figure 6: Comparative performance metrics (Dice Similarity Coefficient) for SAM1 and SAM2 across different model architectures and prompt types.

Key Insights

  • Efficiency: The model's internal memory of anatomical shapes is remarkably robust, often outperforming sparse human prompts.
  • Model Scaling: Interestingly, the sam2_hiera_base_plus model frequently matched or exceeded the performance of the much larger large variant. This suggests that for specific medical tasks, architectural efficiency may be more important than raw parameter count.
  • Prompt Robustness: While single-point prompts occasionally led to "foreground leakage" (selecting the entire leg), the use of negative points or bounding boxes effectively constrained the model to the target anatomy.

Conclusion

This study demonstrates that SAM2 can be effectively adapted for zero-shot 3D medical image segmentation by reframing spatial depth as temporal progression. This approach significantly reduces the manual labor required for volumetric analysis while maintaining high accuracy. Future research will focus on fine-tuning SAM2 on medical-specific datasets and expanding the workflow to other modalities such as CT and ultrasound to further validate its generalizability in clinical settings.

Summary updated

Abstract

Medical image segmentation models often suffer from poor generalizability and annotation costs. We propose an adaptation to Segment Anything Model 2 (SAM2) that leverages its video predictor and memory component to segment 3D medical images with no training and minimal prompting. When segmenting the tibia bone in knee MRI, the model achieves a median Dice similarity coefficient of 0.9643 with only one prompt per volume. We also present comparisons across different model sizes, prompt schemes, and the previous SAM1 model.
BibTeX
@misc{yu2024novel,
  author = {Andrew Seohwan Yu and Mohsen Hariri and Xuecen Zhang and Mingrui Yang and Vipin Chaudhary and Xiaojuan Li},
  title = {{Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2}},
  year = {2024},
  eprint = {2408.04762},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.04762},
}
Figure