MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation

Xiaolong Deng, Huisi Wu, Runhao Zeng, Jing QinCVPR 2024 · DOI

MemSAM adapts the Segment Anything Model (SAM) for echocardiography video segmentation by introducing a space-time memory mechanism. This provides temporal-aware and noise-resilient prompting, achieving state-of-the-art performance on ultrasound videos with minimal human interaction—specifically, a single point prompt on the first frame. This approach effectively bridges the gap between general-purpose foundation models and specialized medical imaging requirements.

The Challenge of Echocardiography

Echocardiography is a cornerstone for diagnosing cardiovascular disease, yet automating its analysis remains challenging due to three primary factors:

  1. Massive Speckle Noise: Grainy artifacts inherent in ultrasound physics obscure anatomical details.
  2. Ambiguous Boundaries: Cardiac structures often lack sharp edges, complicating the distinction between the endocardium and surrounding tissue.
  3. Temporal Variation: The heart's shape and scale change rapidly throughout the cardiac cycle.

While the Segment Anything Model (SAM) has set new standards in general image segmentation, it falters in medical video contexts. Standard SAM lacks temporal awareness, leading to inconsistent "flickering" across frames. Moreover, requiring manual prompts for every frame in a clinical video is impractical for real-world workflows.

Methods: Integrating Space-Time Memory

The proposed MemSAM architecture integrates a Space-Time Memory (STM) mechanism with a medical-optimized version of SAM, known as SAMUS.

MemSAM architecture overview showing the sequential processing of frames guided by memory prompts.

Key innovations include:

  • Memory Prompting: Instead of manual per-frame prompts, MemSAM utilizes a memory bank storing spatial and temporal information from previous frames to guide the current frame's segmentation. This ensures the model maintains tracking across the sequence.
  • Memory Reinforcement Mechanism: To mitigate ultrasound noise, a module uses predicted masks to weight and "clean" features before memory storage, emphasizing cardiac structures over background noise.
  • Sequential Processing: The model processes video frames sequentially. A simple point prompt is required only for the initial frame; all subsequent frames are guided entirely by internal memory prompts.
  • Architecture: MemSAM uses a Vision Transformer (ViT) backbone with "adapters"—small trainable layers—to fine-tune the foundation model for ultrasound textures without retraining the entire SAM network.

Datasets and Evaluation

The model was evaluated on two major public benchmarks:

  1. CAMUS Dataset: 500 cases featuring apical two-chamber and four-chamber views. Testing included both "Full" (all frames annotated) and "Semi" (two frames per video annotated) versions to assess efficiency with sparse data.
  2. EchoNet-Dynamic: A large-scale dataset containing over 10,000 apical two-chamber videos.
Comparison of different prompting and segmentation results on echocardiography frames.

Key Insights

  • Features over Masks: Unlike models that simply propagate masks, MemSAM propagates high-level semantic features. This prevents "error drift," where minor inaccuracies in early frames escalate into total failure later in the sequence.
  • Taming the Foundation Model: The research demonstrates that medical models need not be built from scratch. By "taming" SAM with a temporal memory bank, its powerful representational capabilities are leveraged while adding necessary domain-specific temporal consistency.
  • Clinical Relevance: Beyond pixel accuracy, the authors evaluated the Left Ventricular Ejection Fraction (LVEF). MemSAM's automated measurements showed significantly higher correlation with expert physician measurements compared to previous methods.

Results and Performance

  • Segmentation Accuracy: MemSAM achieved a mean Dice (mDice) score of 93.31% on CAMUS-Semi and 92.78% on EchoNet-Dynamic, outperforming traditional UNet and other SAM adaptations like MedSAM.
  • Clinical Metrics: In LVEF estimation, MemSAM reached a Pearson correlation of 78.92%, a substantial improvement over the next best model (SAMUS at 67.55%).
  • Prompt Efficiency: Ablation studies confirmed robustness; increasing initial point prompts from 1 to 10 yielded negligible gains, proving a single click is sufficient for high-quality tracking.

Limitations and Future Work

The primary limitation is initialization sensitivity. If the first frame is of extremely poor quality (e.g., signal dropout), the memory may initialize incorrectly, causing persistent errors throughout the video. Future work will focus on developing robust automated initialization, reducing computational costs for real-time use, and extending the framework to other medical video modalities like endoscopy or surgical video.

Summary updated

BibTeX
@inproceedings{deng2024memsam,
  author = {Xiaolong Deng and Huisi Wu and Runhao Zeng and Jing Qin},
  title = {{MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00919},
}
Figure