Segment anything model for medical image segmentation: Current applications and future directions

Yichi Zhang, Zhenrong Shen, Rushi Jiao

TL;DR

The Segment Anything Model (SAM) has revolutionized natural image segmentation, but its direct application to medical imaging is hindered by domain differences. This paper surveys recent efforts to adapt SAM for medical segmentation, exploring zero-shot performance, fine-tuning strategies, and novel architectures, highlighting its potential and challenges in this critical domain.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis and treatment, but traditional deep learning models often struggle with generalization across different modalities and tasks. The advent of large-scale foundation models, like SAM, offers a promising path towards more versatile segmentation. However, SAM, trained primarily on natural images, faces challenges when applied to the unique characteristics of medical images (e.g., structural complexity, low contrast, varied modalities). This paper addresses the question of how effectively SAM can be adapted and utilized for medical image segmentation, bridging the gap between its impressive performance on natural images and its current limitations in the medical domain.

Methods

The paper categorizes recent research into two main directions:

  • Zero-shot Evaluation: This involves directly applying SAM to various medical imaging tasks without any fine-tuning. Studies evaluate its performance across different modalities (CT, MRI, X-ray, pathology, etc.) and prompt types (points, boxes, masks).
  • Adaptation Strategies: This encompasses various methods to improve SAM's performance on medical data:
  • Fine-tuning: This includes full fine-tuning of SAM's parameters on medical datasets and parameter-efficient fine-tuning (PEFT) techniques like LoRA (Low-Rank Adaptation) to reduce computational costs.
  • Auto-prompting: Developing mechanisms to automatically generate prompts for SAM, reducing the need for manual expert input. This includes prompt generation from bounding boxes and even text-based prompts.
  • Framework Modification: Adapting SAM's architecture for 3D medical images (e.g., by introducing 3D convolutional layers or attention mechanisms) or integrating it as a component within larger medical segmentation frameworks.
  • Uncertainty Estimation: Enhancing the reliability of SAM's segmentation by estimating and utilizing uncertainty maps to rectify potential errors, especially in auto-prompting scenarios.

Datasets

The paper references numerous datasets used for evaluation, including:

  • AMOS22 Abdominal CT Organ Segmentation dataset: Used for evaluating SAM's zero-shot performance on CT scans.
  • Multi-phase liver tumor segmentation datasets: Used for assessing SAM's performance on tumor segmentation.
  • Brain scans (various qualities, sequences, and lesions): Used for evaluating SAM's brain extraction capabilities.
  • Whole slide imaging (WSI) data: Used for pathological image segmentation.
  • Colonoscopy image datasets: Used for polyp segmentation.
  • Endoscopic surgical instrument segmentation datasets (EndoVis17, EndoVis18): Used for evaluating SAM on surgical instrument segmentation.
  • COSMOS 1050K: A large-scale dataset curated by Huang et al. [12] featuring 18 modalities, 84 objects, and over 1 million images, used for comprehensive evaluation.
  • SA-Med2D-20M Dataset [87]: A large-scale 2D medical image segmentation dataset with 4.6 million images and 19.7 million masks.
  • Various public 3D medical datasets: Used for evaluating 3D adaptation methods.
  • Ultrasound datasets: Used for evaluating SAMUS.
  • CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation datasets: Used for evaluating MA-SAM.
  • Colon and pancreas tumor segmentation datasets: Used for evaluating ProMISe.
  • Unlabeled CT volumes and labeled ones: Used for SegVol.

Key Insights and Ideas

  • Domain Gap is Significant: SAM's performance on medical images, especially in zero-shot settings, is highly variable and often falls short of task-specific models due to fundamental differences between natural and medical image characteristics.
  • Prompt Engineering Matters: While SAM is designed for promptable segmentation, the effectiveness of prompts (points, boxes, scribbles) varies greatly in medical contexts. Box prompts often perform better than point prompts, and manual prompts generally outperform automated ones.
  • Adaptation is Key: Direct application is insufficient; fine-tuning, architectural modifications (especially for 3D), and auto-prompting strategies are crucial for achieving competitive performance in medical image segmentation.
  • Uncertainty is a Valuable Signal: Estimating and utilizing uncertainty in SAM's predictions can significantly improve the reliability and robustness of auto-prompted segmentation, helping to identify and correct errors.
  • 3D is a Major Hurdle: Adapting SAM's 2D architecture to effectively process 3D volumetric medical data is a significant research area, requiring specialized techniques to capture spatial context.
  • Foundation Models as Tools, Not Magic Bullets: SAM can accelerate annotation and serve as a valuable component in medical AI pipelines, but it's not a plug-and-play solution for all medical segmentation tasks without careful adaptation.

Key Results

  • Zero-shot Performance: Varies widely. For instance, Roy et al. [44] found SAM underperformed SOTA by 20.3-40.9% with point prompts on CT, while Hu et al. [46] showed performance improved with more prompt points for tumor segmentation, though still lagging behind U-Net.
  • Fine-tuning Improvements: MedSAM [13] achieved median DSC scores of 94.0% (intracranial hemorrhage CT), 94.4% (glioma MR T1), 81.5% (pneumothorax CXR), and 98.4% (polyp endoscopy), surpassing U-Net.
  • Parameter-Efficient Fine-tuning: Med-SA [14] showed superiority over SAM and previous SOTA methods on 17 medical image segmentation tasks. SAMed [68] achieved a DSC score of 81.88% on par with SOTA methods.
  • 3D Adaptation: 3DSAM-adapter [16] outperformed nnU-Net by 8.25% (kidney tumor), 29.87% (pancreas tumor), and 10.11% (colon cancer). MA-SAM [22] surpassed nnU-Net by 0.9-9.9% in DSC scores on various 3D datasets.
  • Auto-prompting: DeSAM [15] improved the robustness of SAM's fully automatic mode by an average DSC score of 8.96% in dealing with domain shifts. UR-SAM [78] improved segmentation performance by up to 10.7% and 13.8% in DSC scores without manual prompts.

Limitations and Future Work

  • Limitations:
    • Direct application of SAM often yields suboptimal results, especially for objects with weak boundaries, low contrast, small sizes, or irregular shapes.
  • SAM's performance is highly dependent on the quality and type of prompts.
  • Adapting SAM's 2D architecture to 3D medical images remains a challenge, often requiring significant modifications.
  • The domain gap between natural and medical images is substantial, limiting SAM's zero-shot generalization.
  • Future Work:
    • Building Large-Scale Medical Datasets: Continued efforts are needed to create diverse, high-quality medical datasets to better train and evaluate foundation models.
  • Accelerating Medical Image Annotation: Leveraging SAM to speed up the manual annotation process through interactive tools and semi-automatic workflows.
  • Incorporating Diverse Prompts: Exploring more sophisticated prompt types, such as scribbles and text, to inject richer clinical knowledge and improve segmentation accuracy.
  • Towards Multi-Modal Medical Images: Extending SAM to effectively handle and integrate information from multiple medical imaging modalities.
  • Assisting in More Clinical Applications: Integrating SAM into various clinical workflows beyond segmentation, such as surgical guidance and early disease detection.
  • Enhancing Reliability and Robustness: Further research into uncertainty estimation and other techniques to ensure SAM's dependable performance in critical clinical settings.

Summary updated

BibTeX
@article{zhang2024segment,
  author = {Yichi Zhang and Zhenrong Shen and Rushi Jiao},
  title = {{Segment anything model for medical image segmentation: Current applications and future directions}},
  journal = {Computers in Biology and Medicine},
  year = {2024},
  doi = {10.1016/j.compbiomed.2024.108238},
}

Loading related content...

Figure