Cardiac Magnetic Resonance 2D+T Short- and Long-axis Segmentation via Spatio-temporal SAM Adaptation

Zhennong Chen, Sekeun Kim, Hui Ren, Quanzheng Li, Xiang Li

Abstract

Accurate myocardium segmentation across all phases in one cardiac cycle in cine cardiac magnetic resonance (CMR) scans is crucial for comprehensively cardiac function analysis. Despite advancements in deep learning (DL) for automatic cine CMR segmentation, generalizability on unseen data remains a significant challenge. Recently, the segment-anything-model (SAM) has been invented as a segmentation foundation model, known for its accurate segmentation and more importantly, zero-shot generalization. SAM was trained on two-dimensional (2D) natural images; to adapt it for comprehensive cine CMR segmentation, we propose cineCMR-SAM which incorporates both temporal and spatial information through a modified model architecture. Compared to other state-of-the-art (SOTA) methods, our model achieved superior data-specific model segmentation accuracy on the STACOM2011 when fine-tuned on this dataset and demonstrated superior zero-shot generalization on two other large public datasets (ACDC and M&Ms) unseen during fine-tuning. Additionally, we introduced a text prompt feature in cineCMR-SAM to specify the view type of input slices (short-axis or long-axis), enhancing performance across all view types.

Introduction

Automating the segmentation of the myocardium in cine Cardiac Magnetic Resonance (CMR) is a cornerstone of modern cardiac disease diagnosis. However, current deep learning models face a persistent "generalizability gap," often failing when applied to images from different medical centers or scanner vendors (e.g., Siemens vs. GE). Furthermore, cine CMR is inherently 2D+Time (2D+T); models must maintain temporal consistency across the cardiac cycle to be clinically useful. Finally, while most tools focus on Short-Axis (SAX) views, Long-Axis (LAX) views are equally critical for metrics like longitudinal strain but are frequently neglected in automated pipelines.

Methodology: The cineCMR-SAM Architecture

The proposed cineCMR-SAM builds upon the Vision Transformer (ViT) backbone of Meta’s Segment Anything Model (SAM). To bridge the gap between static 2D natural images and dynamic 2D+T medical sequences, the authors introduced several key architectural innovations.

The cineCMR-SAM architecture featuring temporal-spatial adaptation and text prompting.

Temporal-Spatial Adaptation

Inspired by video processing architectures, the model utilizes modified ViT blocks that apply "Time-Space Self-Attention." This sequentially processes temporal dynamics across the cardiac cycle followed by spatial features within each slice. This allows the model to "understand" heart motion over time.

U-Net and Text Prompt Integration

To capture fine-grained anatomical details, refined SAM blocks are embedded within a 2D U-Net framework. Multi-Head Cross-Attention (MHCA) is used in the upsampling path to filter noise. Additionally, the model leverages SAM’s promptable nature by introducing a text encoder. By providing simple prompts like "SAX" or "LAX," the model adjusts its internal logic based on the specific anatomical view.

Parameter-Efficient Fine-Tuning

To preserve the extensive knowledge SAM gained from its original 1-billion-image training set, the authors employed a parameter-efficient strategy: freezing the original spatial weights while training only the new temporal modules, adapters, and U-Net layers.

Experimental Setup

The model was trained and fine-tuned on the STACOM 2011 dataset (100 patients, multi-center, multi-vendor). To test its robustness, zero-shot evaluations were performed on two external datasets:

  • ACDC: 100 cases from a single center.
  • M&Ms (Multi-Centre, Multi-Vendor & Multi-Disease): 136 cases from 5 centers and 4 vendors, serving as the ultimate test for cross-vendor generalizability.

Results and Discussion

cineCMR-SAM demonstrated superior performance and robustness compared to specialized medical imaging models like nnU-Net and U-transformer.

Quantitative comparison of cineCMR-SAM against SOTA methods on ACDC and M&Ms datasets.

Key Findings

  • Superior Accuracy: On the STACOM dataset, the model achieved a Dice score of 0.890. More importantly, it maintained high performance in zero-shot scenarios, achieving Dice scores of 0.850 on ACDC and 0.835 on M&Ms.
  • Prompt Effectiveness: The inclusion of text prompts improved the Dice score for LAX views from 0.837 to 0.845 and significantly reduced boundary errors (Hausdorff Distance).
  • Vendor Agnostic Performance: Performance remained consistent across GE, Philips, Siemens, and Canon scanners, proving the model's resilience to variations in image acquisition.

Conclusion

This study highlights the power of adapting foundation models for specialized medical tasks. By incorporating temporal-spatial attention and text-based context, cineCMR-SAM overcomes the traditional limitations of medical DL models regarding generalizability and view diversity. Future work will focus on automating the prompting process and incorporating anatomical priors to ensure segmentations remain physiologically consistent.

Summary updated

Abstract

Accurate myocardium segmentation across all phases in one cardiac cycle in cine cardiac magnetic resonance (CMR) scans is crucial for comprehensively cardiac function analysis. Despite advancements in deep learning (DL) for automatic cine CMR segmentation, generalizability on unseen data remains a significant challenge. Recently, the segment-anything-model (SAM) has been invented as a segmentation foundation model, known for its accurate segmentation and more importantly, zero-shot generalization. SAM was trained on two-dimensional (2D) natural images; to adapt it for comprehensive cine CMR segmentation, we propose cineCMR-SAM which incorporates both temporal and spatial information through a modified model architecture. Compared to other state-of-the-art (SOTA) methods, our model achieved superior data-specific model segmentation accuracy on the STACOM2011 when fine-tuned on this dataset and demonstrated superior zero-shot generalization on two other large public datasets (ACDC and M&Ms) unseen during fine-tuning. Additionally, we introduced a text prompt feature in cineCMR-SAM to specify the view type of input slices (short-axis or long-axis), enhancing performance across all view types.
BibTeX
@inproceedings{chen2024cardiac,
  author = {Zhennong Chen and Sekeun Kim and Hui Ren and Quanzheng Li and Xiang Li},
  title = {{Cardiac Magnetic Resonance 2D+T Short- and Long-axis Segmentation via Spatio-temporal SAM Adaptation}},
  booktitle = {MedAGI @ MICCAI},
  year = {2024},
  url = {https://openreview.net/forum?id=12pUnJwore},
}
Figure