SemiSAM: Exploring SAM for Enhancing Semi-Supervised Medical Image Segmentation with Extremely Limited Annotations

Yichi Zhang, Yuan Cheng, Yuan Qi

Abstract

Semi-supervised learning has attracted much attention due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods, which is especially important for medical image segmentation which typically requires intensive pixel/voxel-wise labeling by domain experts. Although semi-supervised methods can improve the performance by utilizing unlabeled data, there are still gaps between fully supervised methods under extremely limited annotation scenarios. In this paper, we propose a simple yet efficient strategy to explore the usage of the Segment Anything Model (SAM) for enhancing semi-supervised medical image segmentation. Concretely, the segmentation model trained with domain knowledge provides information for localization and generating input prompts to the SAM. Then the generated pseudo-labels of SAM are utilized as additional supervision to assist in the learning procedure of the semi-supervised framework. Extensive experiments demonstrate that SemiSAM significantly improves the performance of existing semi-supervised frameworks when only one or a few labeled images are available and shows strong efficiency as a plug-and-play strategy for semi-supervised medical image segmentation.

Introduction

Medical image segmentation is a critical task in radiology, yet training robust AI models typically requires extensive expert annotations. Semi-supervised learning (SSL) addresses this "data bottleneck" by leveraging a small set of labeled data alongside a larger pool of unlabeled images. However, when labels are extremely scarce—such as having only one or two labeled patients—traditional SSL methods like the "Mean Teacher" approach often fail. In these scenarios, the teacher model, trained on insufficient data, propagates errors to the student model. SemiSAM introduces a novel approach to bridge this gap by incorporating the Segment Anything Model (SAM) as a source of external knowledge to guide the training process.

Methodology

The SemiSAM framework is designed as a plug-and-play strategy that augments standard SSL architectures with a SAM-assisted branch. The system consists of three primary components:

  1. Student Model: A standard segmentation network (e.g., V-Net) that processes input images to generate initial predictions.
  2. Teacher Model: An ensemble of the student's previous iterations, providing consistency targets for unlabeled data via Exponential Moving Average (EMA).
  3. SAM-Assisted Branch: This branch utilizes the student model's coarse predictions to generate prompts for SAM-Med3D. An uncertainty-aware strategy identifies high-confidence regions to serve as point prompts. SAM then generates a refined mask, which acts as an additional supervision signal ($L_{sam}$).
The SemiSAM framework integrates the Segment Anything Model (SAM) into a semi-supervised learning (SSL) architecture.

The total loss function combines supervised loss on labeled data, consistency loss between student and teacher, and the SAM-assisted consistency loss. A "ramp-down" strategy is employed for the SAM loss, allowing it to provide strong guidance early in training while letting the domain-specific student model take over as it matures.

Experimental Results

The method was evaluated on the Left Atrium (LA) Segmentation Challenge Dataset using 100 3D MRIs. Experiments focused on extremely limited label scenarios (1, 2, 4, and 8 labeled cases).

SemiSAM demonstrated significant quantitative improvements over the Mean Teacher (MT) baseline. With only 1 labeled case, the Dice score improved from 30.60% to 41.38%. With 2 labeled cases, it rose from 58.17% to 69.46%, and with 4 cases, it reached 80.42% compared to the 72.40% baseline.

Qualitatively, the impact of SAM-assisted refinement is evident in the 3D reconstructions. As shown in Figure 2, SemiSAM produces smoother and more anatomically coherent segmentations compared to models trained without it, which often exhibit fragmented or noisy outputs in low-data regimes.

Qualitative comparison of segmentation results on the LA dataset. SemiSAM produces smoother, more accurate 3D reconstructions compared to the baseline.

Discussion and Conclusion

SemiSAM effectively utilizes SAM as a refiner rather than a standalone segmenter. By using a domain-specific model to "prompt" SAM, the framework overcomes SAM's inherent limitations with medical textures while benefiting from its generalizable boundary knowledge. While the benefits of SemiSAM are most pronounced in extremely low-data scenarios, its performance converges with standard methods as more labeled data becomes available. Future work will explore the integration of newer foundation models like SAM 2 and broader applications across diverse medical imaging modalities.

Summary updated

Abstract

Semi-supervised learning has attracted much attention due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods, which is especially important for medical image segmentation which typically requires intensive pixel/voxel-wise labeling by domain experts. Although semi-supervised methods can improve the performance by utilizing unlabeled data, there are still gaps between fully supervised methods under extremely limited annotation scenarios. In this paper, we propose a simple yet efficient strategy to explore the usage of the Segment Anything Model (SAM) for enhancing semi-supervised medical image segmentation. Concretely, the segmentation model trained with domain knowledge provides information for localization and generating input prompts to the SAM. Then the generated pseudo-labels of SAM are utilized as additional supervision to assist in the learning procedure of the semi-supervised framework. Extensive experiments demonstrate that SemiSAM significantly improves the performance of existing semi-supervised frameworks when only one or a few labeled images are available and shows strong efficiency as a plug-and-play strategy for semi-supervised medical image segmentation.
BibTeX
@inproceedings{zhang2024semisam,
  author = {Yichi Zhang and Yuan Cheng and Yuan Qi},
  title = {{SemiSAM: Exploring SAM for Enhancing Semi-Supervised Medical Image Segmentation with Extremely Limited Annotations}},
  booktitle = {BIBM},
  year = {2024},
  url = {https://openreview.net/forum?id=mxFkEW0rfb},
}
Figure