SemiSAM+: Rethinking semi-supervised medical image segmentation in the era of foundation models

Yichi Zhang, Bohao Lv, Le Xue, Wenbo Zhang, Yuchen Liu, Yu Fu, Yuan Cheng, Yuan Qi

TL;DR

SemiSAM+ is a novel semi-supervised learning framework that leverages the "universal" knowledge of foundation models (like SAM) to train medical image segmentation models with minimal human intervention. By using a task-specific "specialist" to automatically prompt a frozen "generalist" foundation model, it achieves state-of-the-art performance even when only a single labeled image is available.

What Problem Does This Paper Address?

Deep learning has revolutionized medical image segmentation, but it has a "data hunger" problem. Training these models typically requires massive amounts of pixel-perfect labels created by highly trained radiologists—a process that is expensive, slow, and often bottlenecked by privacy regulations.

Semi-supervised learning (SSL) tries to fix this by using a small amount of labeled data alongside a large pool of unlabeled data. However, traditional SSL methods often "collapse" or perform poorly when the number of labels is extremely low (e.g., only 1 to 5 cases). Meanwhile, new "Foundation Models" like the Segment Anything Model (SAM) have incredible general segmentation abilities but often struggle with the specific textures of medical scans or require tedious manual prompting for every new image. SemiSAM+ aims to bridge this gap: using foundation models to provide high-quality "pseudo-labels" to train specialist models automatically.

Methods

The authors propose a Specialist-Generalist Collaborative Learning architecture:

  1. The Specialist Model: A trainable, task-specific model (like a 3D U-Net). Its job is to learn the specific anatomy of the target (e.g., a heart or a tumor).
  2. The Generalist Model: A frozen, pre-trained foundation model (specifically SAM-Med3D). It possesses "broad" knowledge of what objects look like but needs a "hint" (a prompt) to know what to segment.
  3. The Collaborative Loop: During training on unlabeled data, the Specialist generates a coarse initial prediction. This prediction is automatically converted into positional prompts (points or masks) and fed into the Generalist. The Generalist then produces a refined, high-quality segmentation.
  4. Confidence-Aware Regularization: This is the "plus" in SemiSAM+. The system estimates the uncertainty of the Generalist by giving it slightly different prompts. If the Generalist’s outputs vary wildly, the Specialist is told to ignore that region, preventing it from learning from "noisy" or incorrect pseudo-labels.

Crucially, the Generalist remains frozen, meaning the computational cost during training is kept in check, and the final deployed model is just the lightweight Specialist.

Datasets

The framework was evaluated on three distinct 3D segmentation tasks:

  • Left Atrium (LA): 100 3D gadolinium-enhanced MRIs from the Left Atrium Segmentation Challenge.
  • BraTS 2019: 250 3D MRI scans for brain tumor segmentation.
  • In-house PETS: 100 FDG-PET images focused on whole heart segmentation, used to test the model's flexibility across different imaging modalities.

Key Insights and Ideas

  • The Specialist as a "Prompter": The most brilliant insight is that the specialist model doesn't need to be perfect at segmenting early on; it just needs to be good enough to point the foundation model in the right direction.
  • 3D is Essential: While the original SAM is 2D, the authors show that using a 3D-native foundation model (SAM-Med3D) is vital for medical imaging to maintain spatial consistency across slices.
  • Plug-and-Play Flexibility: SemiSAM+ isn't a standalone model but a strategy. It can be wrapped around existing SSL methods (like Mean Teacher or Dual Task Consistency) to provide an immediate performance boost.
  • Uncertainty as a Filter: By using multiple prompts to "interrogate" the foundation model, the system can autonomously decide which pseudo-labels are trustworthy, mimicking how a student might double-check a teacher's hint.

Key Results

  • Extreme Data Scarcity: On the Left Atrium dataset with only 1 labeled case, SemiSAM+ achieved a Dice score of 48.91%, significantly outperforming the standard SSL baseline of 40.64%.
  • Scaling Up: With 5 labeled cases, SemiSAM+ reached 77.02% Dice, approaching the performance of models trained on much larger datasets.
  • State-of-the-Art: When combined with advanced SSL backbones (like SDCL), SemiSAM+ hit 90.14% Dice on the LA dataset with 5 labels, outperforming all other compared semi-supervised and SAM-based fine-tuning methods.
  • Cross-Modality: The PETS dataset experiments proved the framework works even on PET scans, a modality the foundation model wasn't originally specialized for, by combining multiple generalists (SAM-Med3D + SegAnyPET).

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Single-Target Focus: The current iteration is optimized for segmenting one specific structure at a time. Adapting this to multi-organ segmentation (e.g., segmenting all abdominal organs at once) would require more complex prompting strategies.
  • Complex Shapes: Point-based prompts can struggle with "non-compact" or highly irregular targets (like thin, branching vessels).
  • Future Directions: The team plans to explore more flexible prompt types, such as "scribbles" or semantic text prompts (e.g., "segment the tumor near the lateral ventricle"), to make the interaction between specialist and generalist even more robust.

Summary updated

Abstract

Deep learning-based medical image segmentation typically requires large amount of labeled data for training, making it less applicable in clinical settings due to high annotation cost. Semi-supervised learning (SSL) has emerged as an appealing strategy due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods. Beyond existing model-centric advancements of SSL by designing novel regularization strategies, we anticipate a paradigmatic shift due to the emergence of promptable segmentation foundation models with universal segmentation capabilities using positional prompts represented by Segment Anything Model (SAM). In this paper, we present SemiSAM+, a foundation model-driven SSL framework to efficiently learn from limited labeled data for medical image segmentation. SemiSAM+ consists of one or multiple promptable foundation models as generalist models, and a trainable task-specific segmentation model as specialist model. For a given new segmentation task, the training is based on the specialist-generalist collaborative learning procedure, where the trainable specialist model delivers positional prompts to interact with the frozen generalist models to acquire pseudo-labels, and then the generalist model output provides the specialist model with informative and efficient supervision which benefits the automatic segmentation and prompt generation in turn. Extensive experiments on two public datasets and one in-house clinical dataset demonstrate that SemiSAM+ achieves significant performance improvement, especially under extremely limited annotation scenarios, and shows strong efficiency as a plug-and-play strategy that can be easily adapted to different specialist and generalist models.
BibTeX
@article{zhang2025semisam,
  author = {Yichi Zhang and Bohao Lv and Le Xue and Wenbo Zhang and Yuchen Liu and Yu Fu and Yuan Cheng and Yuan Qi},
  title = {{SemiSAM+: Rethinking semi-supervised medical image segmentation in the era of foundation models}},
  journal = {Medical Image Analysis},
  year = {2025},
  doi = {10.1016/j.media.2025.103733},
}

Loading related content...

Figure