Temporal-spatial Adaptation of Promptable SAM Enhance Accuracy and Generalizability of cine CMR Segmentation

Zhennong Chen, Sekeun Kim, Hui Ren, Quanzheng Li, Xiang Li

Abstract

Accurate myocardium segmentation across all phases in one cardiac cycle in cine cardiac magnetic resonance (CMR) scans is crucial for comprehensively cardiac function analysis. Despite advancements in deep learning (DL) for automatic cine CMR segmentation, generalizability on unseen data remains a significant challenge. Recently, the segment-anything-model (SAM) has been invented as a segmentation foundation model, known for its accurate segmentation and more importantly, zero-shot generalization. SAM was trained on two-dimensional (2D) natural images; to adapt it for comprehensive cine CMR segmentation, we propose cineCMR-SAM which incorporates both temporal and spatial information through a modified model architecture. Compared to other state-of-the-art (SOTA) methods, our model achieved superior data-specific model segmentation accuracy on the STACOM2011 when fine-tuned on this dataset and demonstrated superior zero-shot generalization on two other large public datasets (ACDC and M&Ms) unseen during fine-tuning. Additionally, we introduced a text prompt feature in cineCMR-SAM to specify the view type of input slices (short-axis or long-axis), enhancing performance across all view types.

Introduction

Accurate segmentation of the heart muscle (myocardium) throughout the entire cardiac cycle is essential for assessing heart health and calculating clinical parameters like ejection fraction. While deep learning (DL) has automated much of this process, two major hurdles persist: generalizability and temporal consistency. Models trained on data from one hospital or scanner vendor often perform poorly when applied to data from another. Furthermore, cine CMR is essentially a video; standard 2D models often ignore the relationship between frames, leading to inconsistent segmentations over time.

To address these issues, this paper introduces cineCMR-SAM, a specialized adaptation of the Segment Anything Model (SAM). By leveraging SAM's foundation as a robust zero-shot segmenter and augmenting it with temporal-spatial attention and text-based view prompts, the authors significantly improve the model's ability to generalize across different medical centers and scanner vendors.

Proposed Method: cineCMR-SAM

The authors propose a modified SAM architecture that integrates temporal-spatial adaptation and a U-Net hybrid structure to handle the complexities of 2D+T cardiac imaging.

The cineCMR-SAM architecture featuring temporal-spatial adaptation blocks and text prompt integration.

Key architectural modifications include:

  • Temporal-Spatial Adaptation: Inspired by the "TimeSformer" architecture, the Vision Transformer (ViT) blocks were refined to apply "temporal self-attention" (looking at the same spot across time) followed by "spatial self-attention" (looking at the whole frame).
  • U-Net Hybrid Structure: These refined blocks are embedded into a U-Net framework, allowing the model to capture fine-grained local details via skip connections while maintaining global context.
  • Text Prompts for View Type: A text prompt feature allows users to input "SAX" (short-axis) or "LAX" (long-axis), helping the model adjust its segmentation strategy based on the heart's orientation.
  • Parameter-Efficient Fine-Tuning: To preserve SAM's pre-trained knowledge, original spatial weights were frozen, and only the new temporal blocks, adapters, and U-Net layers were trained.

Experimental Setup and Datasets

The study utilized three major public datasets to evaluate performance and generalization:

  1. STACOM2011: Used for initial fine-tuning, containing 100 patients with both SAX and LAX views.
  2. ACDC: A single-center dataset with 100 cases used for zero-shot testing.
  3. M&Ms (Multi-Centre, Multi-Vendor & Multi-Disease): A challenging zero-shot test set with 136 cases from five centers and four different scanner vendors (GE, Philips, Siemens, Canon).

Key Results and Discussion

The cineCMR-SAM model demonstrated superior accuracy and robust generalization across all tested datasets.

Qualitative segmentation results across ACDC, M&Ms, and STACOM-LAX datasets, showing high consistency with ground truth.
  • Superior Accuracy: On the STACOM dataset, cineCMR-SAM achieved a Dice score of 0.890, outperforming nnU-Net (0.877).
  • Robust Generalization: In zero-shot testing on the M&Ms dataset, the model maintained a Dice score of 0.835, significantly higher than the U-transformer (0.810).
  • Prompt Impact: Adding the text prompt "LAX" improved the Dice score for long-axis views from 0.837 to 0.845 and reduced boundary error (Hausdorff Distance) from 6.264 to 5.440 pixels.
  • Vendor Agnostic Performance: The model performed consistently across different scanner types, including vendors like Canon that were not present in the training data.

Conclusion and Future Work

This study proves that adapting a massive natural-image foundation model like SAM provides a significant "generalization buffer" for medical imaging tasks. By treating the cardiac cycle as a sequence rather than independent slices, cineCMR-SAM maintains anatomical consistency across the heartbeat.

Future work will focus on automating the view-type prompts, exploring more granular prompts (e.g., "Basal SAX"), and incorporating cardiac physical priors to ensure realistic volume changes. The authors are also working on clinical validation using in-house datasets to calculate parameters like ejection fraction.

Summary updated

Abstract

Accurate myocardium segmentation across all phases in one cardiac cycle in cine cardiac magnetic resonance (CMR) scans is crucial for comprehensively cardiac function analysis. Despite advancements in deep learning (DL) for automatic cine CMR segmentation, generalizability on unseen data remains a significant challenge. Recently, the segment-anything-model (SAM) has been invented as a segmentation foundation model, known for its accurate segmentation and more importantly, zero-shot generalization. SAM was trained on two-dimensional (2D) natural images; to adapt it for comprehensive cine CMR segmentation, we propose cineCMR-SAM which incorporates both temporal and spatial information through a modified model architecture. Compared to other state-of-the-art (SOTA) methods, our model achieved superior data-specific model segmentation accuracy on the STACOM2011 when fine-tuned on this dataset and demonstrated superior zero-shot generalization on two other large public datasets (ACDC and M&Ms) unseen during fine-tuning. Additionally, we introduced a text prompt feature in cineCMR-SAM to specify the view type of input slices (short-axis or long-axis), enhancing performance across all view types.
BibTeX
@inproceedings{chen2024temporal,
  author = {Zhennong Chen and Sekeun Kim and Hui Ren and Quanzheng Li and Xiang Li},
  title = {{Temporal-spatial Adaptation of Promptable SAM Enhance Accuracy and Generalizability of cine CMR Segmentation}},
  booktitle = {MedAGI @ MICCAI},
  year = {2024},
  url = {https://openreview.net/forum?id=12pUnJwore},
}
Figure