Research on Medical Image Segmentation Based on SAM and Its Future Prospects

Kebin Fan, Liang Liang, Hao Li, Weijun Situ, Wei Zhao, Ge LiBioengineering 2025 · DOI

TL;DR

The Segment Anything Model (SAM) has shown remarkable versatility in natural image segmentation, but its direct application to medical imaging is challenging due to domain differences. This review surveys recent research adapting SAM for medical tasks, highlighting its potential and the ongoing efforts to overcome its limitations for clinical use.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis and treatment planning, but traditional deep learning models require task-specific training, demanding extensive labeled data for each new segmentation task. The Segment Anything Model (SAM) offers a promising alternative with its promptable, zero-shot segmentation capabilities, potentially reducing annotation burdens. However, the significant differences between natural and medical images (e.g., intensity variations, noise, anatomical complexity) raise questions about SAM's direct applicability and robustness in the medical domain. This paper addresses this gap by reviewing current research on adapting SAM for medical image segmentation, evaluating its performance, and exploring future directions.

Methods

The paper discusses various approaches to adapt SAM for medical image segmentation:

  • Zero-Shot Evaluation: Initial studies directly applied SAM to medical datasets using different prompting strategies (e.g., points, bounding boxes) to assess its out-of-the-box performance.
  • Fine-Tuning:
    • Full-Parameter Fine-Tuning: Adapting all of SAM's parameters to medical data, showing significant performance gains but requiring substantial computational resources.
  • Parameter-Efficient Fine-Tuning (PEFT): Techniques like LoRA (Low-Rank Adaptation) are used to fine-tune only a small fraction of SAM's parameters, achieving comparable performance with much lower computational cost.
  • Architectural Modifications:
    • Adapters: Introducing specialized modules (e.g., Med-SA, 3DSAM Adapter) to adapt SAM's image encoder or mask decoder for medical data, including 3D volumes.
  • Hybrid Architectures: Combining SAM with established medical segmentation models like U-Net or nnUNet to leverage the strengths of both.
  • Automated Prompt Mechanisms: Developing methods to automatically generate prompts for SAM, reducing the need for manual user interaction. This includes using object detection models (like YOLOv8) or learning prompt generation strategies.
  • Uncertainty Estimation: Incorporating uncertainty quantification to improve SAM's robustness to prompt variations and provide reliable segmentation outputs.
  • 3D Adaptation: Developing specific methods to extend SAM's capabilities from 2D to 3D medical image segmentation, addressing challenges in volumetric data.

Datasets

The paper mentions evaluations on numerous public medical imaging datasets across various modalities and anatomical regions. Specific datasets and benchmarks highlighted include:

  • AMOS22: For abdominal organ segmentation.
  • COSMOS1050K: A large benchmark comprising 52 datasets with 18 modalities and 84 anatomical/pathological targets.
  • Various datasets for skin cancer, polyp, tumor, organ, and anatomical structure segmentation across modalities like CT, MRI, Ultrasound, X-ray, and histopathology.
  • SA-1B: The original dataset used to train SAM, containing 11 million images and 1.1 billion masks (mentioned for comparison with medical dataset scale).

Key Insights and Ideas

  • Domain Shift is Critical: SAM's performance in medical imaging is heavily influenced by image dimensionality, modality, object size, and contrast, often underperforming specialized models without adaptation.
  • Prompt Engineering Matters: While SAM is promptable, its performance is highly sensitive to prompt quality and type. Box prompts generally perform better than point prompts for medical tasks due to richer spatial information.
  • PEFT is the Way Forward for Efficiency: Parameter-efficient fine-tuning methods are crucial for making SAM practical for clinical deployment, offering a good balance between accuracy and computational cost.
  • 3D Adaptation is Essential: Directly applying 2D SAM to 3D medical data is suboptimal; dedicated 3D adaptation techniques are necessary to capture volumetric context.
  • Hybrid Approaches Offer Synergy: Combining SAM with traditional medical segmentation models can leverage SAM's generalization capabilities and the domain-specific accuracy of established methods.
  • Automation is Key for Usability: Developing automated prompt generation and segmentation pipelines is vital for reducing manual effort and increasing clinical adoption.
  • Uncertainty is Informative: Quantifying uncertainty in SAM's predictions can enhance its reliability and provide clinicians with actionable insights.

Key Results

  • Zero-Shot Performance: SAM often underperforms state-of-the-art (SOTA) methods in zero-shot medical segmentation, with Dice Similarity Coefficients (DSCs) declining significantly (e.g., 20.3–40.9% for abdominal organs with point prompts).
  • Fine-Tuning Gains: Full-parameter fine-tuning can improve DSCs substantially (e.g., from 81.25% to 88.79% for skin cancer segmentation).
  • PEFT Effectiveness: Methods like Med-SA and SAMed achieve comparable performance to full fine-tuning with minimal parameter updates (e.g., 0.1% of parameters), significantly reducing computational load.
  • 3D Adaptation Success: Models like 3DSAM Adapter have shown substantial improvements over nnU-Net in 3D segmentation tasks (e.g., 8.25% for renal tumors, 29.87% for pancreatic tumors).
  • Hybrid Model Performance: SAM-UNet achieved a DSC of 0.883 on a large pathology dataset, outperforming original SAM with manual prompts.
  • Automated Prompting: Methods like MedLSAM and AutoSAM demonstrate effective segmentation without manual prompts, achieving competitive results.

Limitations and Future Work

  • Limitations:
    • Robustness and Consistency: SAM struggles with consistency across diverse datasets and modalities, particularly with low-contrast images, ambiguous boundaries, and complex structures.
  • Computational Overhead: SAM's generalized architecture leads to higher inference latency and memory costs, limiting real-time applicability.
  • Prompt Sensitivity: Segmentation results are highly dependent on prompt quality, which can lead to diagnostic unreliability.
  • Dataset Scarcity: Creating large, high-quality medical datasets for fine-tuning and benchmarking is time-consuming and expensive compared to natural image datasets.
  • Future Work:
    • Dataset Development: Curating large-scale, domain-specific medical image datasets, especially 3D ones, for better fine-tuning and benchmarking.
  • Hybrid Frameworks: Integrating SAM with lightweight, task-optimized backbones for improved performance on specialized medical tasks.
  • Deployment Optimization: Enhancing clinical usability through latency reduction, model compression, and inference acceleration.
  • Prompt Robustness: Developing reliable prompt engineering protocols and tools for consistent, reproducible segmentation.
  • Domain-Specific Adaptation: Continued research into techniques that bridge the gap between natural image pre-training and medical image requirements.
  • Clinical Feasibility and Ethical Considerations: Addressing computational feasibility, ensuring robust prompt engineering, and establishing clear guidelines for human-AI collaboration.

Summary updated

Abstract

The rapid advancement of prompt-based models in natural language processing and image generation has revolutionized the field of image segmentation. The introduction of the Segment Anything Model (SAM) has further invigorated this domain with its unprecedented versatility. However, its applicability to medical image segmentation remains uncertain due to significant disparities between natural and medical images, which demand careful consideration. This study comprehensively analyzes recent efforts to adapt SAM for medical image segmentation, including empirical benchmarking and methodological refinements aimed at bridging the gap between SAM's capabilities and the unique challenges of medical imaging. Furthermore, we explore future directions for SAM in this field. While direct application of SAM to complex, multimodal, and multi-target medical datasets may not yet yield optimal results, insights from these efforts provide crucial guidance for developing foundational models tailored to the intricacies of medical image analysis. Despite existing challenges, SAM holds considerable potential to demonstrate its unique advantages and robust capabilities in medical image segmentation in the near future.
BibTeX
@inproceedings{fan2025research,
  author = {Kebin Fan and Liang Liang and Hao Li and Weijun Situ and Wei Zhao and Ge Li},
  title = {{Research on Medical Image Segmentation Based on SAM and Its Future Prospects}},
  booktitle = {Bioengineering},
  year = {2025},
  doi = {10.3390/bioengineering12060608},
}
Figure