SemiSAM+: Rethinking Semi-Supervised Medical Image Segmentation in the Era of Foundation Models

Yichi Zhang, Bohao Lv, Le Xue, Wenbo Zhang, Yuchen Liu, Yu Fu, Yuan Cheng, Yuan Qi

Abstract

Deep learning-based medical image segmentation typically requires large amount of labeled data for training, making it less applicable in clinical settings due to high annotation cost. Semi-supervised learning (SSL) has emerged as an appealing strategy due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods. Beyond existing model-centric advancements of SSL by designing novel regularization strategies, we anticipate a paradigmatic shift due to the emergence of promptable segmentation foundation models with universal segmentation capabilities using positional prompts represented by Segment Anything Model (SAM). In this paper, we present SemiSAM+, a foundation model-driven SSL framework to efficiently learn from limited labeled data for medical image segmentation. SemiSAM+ consists of one or multiple promptable foundation models as generalist models, and a trainable task-specific segmentation model as specialist model. For a given new segmentation task, the training is based on the specialist-generalist collaborative learning procedure, where the trainable specialist model delivers positional prompts to interact with the frozen generalist models to acquire pseudo-labels, and then the generalist model output provides the specialist model with informative and efficient supervision which benefits the automatic segmentation and prompt generation in turn. Extensive experiments on two public datasets and one in-house clinical dataset demonstrate that SemiSAM+ achieves significant performance improvement, especially under extremely limited annotation scenarios, and shows strong efficiency as a plug-and-play strategy that can be easily adapted to different specialist and generalist models.

Introduction

Deep learning has revolutionized medical image segmentation, yet it remains constrained by a significant "data hunger" problem. Training robust models typically necessitates vast quantities of pixel-perfect labels generated by expert radiologists—a process that is prohibitively expensive, time-consuming, and often restricted by privacy regulations.

Semi-supervised learning (SSL) addresses this by leveraging a small set of labeled data alongside a larger pool of unlabeled data. However, traditional SSL methods often suffer from performance "collapse" in extremely low-data regimes (e.g., 1 to 5 labeled cases). Concurrently, foundation models like the Segment Anything Model (SAM) have demonstrated remarkable zero-shot segmentation capabilities but often struggle with the specific textures of medical scans or require manual prompting. SemiSAM+ bridges this gap by using foundation models to provide high-quality pseudo-labels, enabling the automatic training of task-specific specialist models with minimal human intervention.

SemiSAM+ addresses the performance gap in extremely limited SSL settings by leveraging the knowledge of pretrained foundation models.

Methodology: Specialist-Generalist Collaborative Learning

The core of SemiSAM+ is a collaborative architecture between two distinct model types:

  1. The Specialist Model: A trainable, task-specific model (e.g., a 3D U-Net) tasked with learning the specific anatomy of the target structure, such as the heart or a tumor.
  2. The Generalist Model: A frozen, pre-trained foundation model (specifically SAM-Med3D). It possesses broad visual knowledge but requires positional prompts to identify specific targets.
  3. The Collaborative Loop: During training on unlabeled data, the Specialist generates a coarse initial prediction. This is automatically converted into positional prompts (points or masks) and fed into the Generalist. The Generalist then produces a refined, high-quality segmentation.
  4. Confidence-Aware Regularization: To handle potential errors, the system estimates the Generalist's uncertainty by providing slightly varied prompts. If the outputs are inconsistent, the Specialist ignores that region, preventing the learning of "noisy" pseudo-labels.

This approach keeps the Generalist frozen, maintaining manageable computational costs during training. The final deployed model is only the lightweight Specialist, ensuring inference efficiency.

Experimental Setup

The framework was rigorously evaluated across three 3D segmentation tasks:

  • Left Atrium (LA): 100 3D gadolinium-enhanced MRIs from the Left Atrium Segmentation Challenge.
  • BraTS 2019: 250 3D MRI scans for brain tumor segmentation.
  • In-house PETS: 100 FDG-PET images for whole heart segmentation, testing cross-modality flexibility.

Results and Discussion

SemiSAM+ demonstrates exceptional performance, particularly in scenarios of extreme data scarcity. On the Left Atrium dataset with only one labeled case, SemiSAM+ achieved a Dice score of 48.91%, significantly outperforming the standard SSL baseline of 40.64%. With five labeled cases, performance reached 77.02% Dice. When integrated with advanced SSL backbones like SDCL, SemiSAM+ achieved 90.14% Dice on the LA dataset with five labels, surpassing existing semi-supervised and SAM-based fine-tuning methods.

Key Insights

  • Automated Prompting: The specialist model acts as an automated "prompter," directing the foundation model's broad knowledge toward specific medical targets.
  • 3D Native Processing: Utilizing a 3D-native foundation model (SAM-Med3D) is crucial for maintaining spatial consistency in medical imaging.
  • Plug-and-Play Integration: SemiSAM+ is a strategy that can be wrapped around existing SSL methods to provide immediate performance gains.
  • Cross-Modality Robustness: Experiments on PET scans showed that combining multiple generalists (e.g., SAM-Med3D + SegAnyPET) allows the framework to adapt to modalities outside the foundation model's original training data.

Conclusion and Future Work

SemiSAM+ represents a paradigmatic shift in SSL by moving from purely model-centric regularization to foundation model-driven supervision. While highly effective, current limitations include a focus on single-target structures and challenges with highly irregular, non-compact shapes. Future research will explore more flexible prompt types, such as scribbles or semantic text prompts, to further enhance the robustness and versatility of the specialist-generalist interaction.

Summary updated

Abstract

Deep learning-based medical image segmentation typically requires large amount of labeled data for training, making it less applicable in clinical settings due to high annotation cost. Semi-supervised learning (SSL) has emerged as an appealing strategy due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods. Beyond existing model-centric advancements of SSL by designing novel regularization strategies, we anticipate a paradigmatic shift due to the emergence of promptable segmentation foundation models with universal segmentation capabilities using positional prompts represented by Segment Anything Model (SAM). In this paper, we present SemiSAM+, a foundation model-driven SSL framework to efficiently learn from limited labeled data for medical image segmentation. SemiSAM+ consists of one or multiple promptable foundation models as generalist models, and a trainable task-specific segmentation model as specialist model. For a given new segmentation task, the training is based on the specialist-generalist collaborative learning procedure, where the trainable specialist model delivers positional prompts to interact with the frozen generalist models to acquire pseudo-labels, and then the generalist model output provides the specialist model with informative and efficient supervision which benefits the automatic segmentation and prompt generation in turn. Extensive experiments on two public datasets and one in-house clinical dataset demonstrate that SemiSAM+ achieves significant performance improvement, especially under extremely limited annotation scenarios, and shows strong efficiency as a plug-and-play strategy that can be easily adapted to different specialist and generalist models.
BibTeX
@article{zhang2025semisam,
  author = {Yichi Zhang and Bohao Lv and Le Xue and Wenbo Zhang and Yuchen Liu and Yu Fu and Yuan Cheng and Yuan Qi},
  title = {{SemiSAM+: Rethinking Semi-Supervised Medical Image Segmentation in the Era of Foundation Models}},
  journal = {Medical Image Analysis},
  year = {2025},
  doi = {10.1016/j.media.2025.103733},
}
Figure