SemiSAM: Enhancing Semi-Supervised Medical Image Segmentation via SAM-Assisted Consistency Regularization

Yichi Zhang, Jin Yang, Yuchen Liu, Yuan Cheng, Yuan Qi

Medical image segmentation—the process of outlining organs or tumors in scans like MRIs and CTs—is a cornerstone of modern radiology. However, training the AI models to do this usually requires thousands of painstaking, hand-drawn annotations from expert doctors. Semi-supervised learning (SSL) tries to solve this by learning from a few labeled images and a mountain of unlabeled ones. But when labels are extremely scarce (think: only one or two labeled patients), even the best SSL models struggle.

A new paper, "SemiSAM: Enhancing Semi-Supervised Medical Image Segmentation via SAM-Assisted Consistency Regularization," introduces a clever way to bridge this gap by using a "foundation model" as a high-level teaching assistant.

TL;DR

SemiSAM integrates the Segment Anything Model (SAM) into semi-supervised training workflows to provide additional guidance. By using a domain-specific model to "prompt" SAM, the framework generates high-quality pseudo-labels that significantly boost performance in scenarios where only 1–4 labeled images are available.

What Problem Does This Paper Address?

The "data bottleneck" is real in medical AI. While we have plenty of unlabeled medical images, getting expert-level pixel-wise labels is expensive and slow. Existing semi-supervised methods often use a "Mean Teacher" approach, where a student model learns from a teacher model's predictions on unlabeled data.

The problem? If the teacher model is only trained on one or two images, it isn't a very good teacher. It makes "similar but incorrect" mistakes, and the student just ends up learning those errors. The authors wanted to find a way to inject "external knowledge" into this loop to keep the training on track when data is at its thinnest.

Methods

The researchers propose SemiSAM, a plug-and-play strategy that adds a third branch to the standard Student-Teacher setup:

  1. The Main Branch (Student): A standard segmentation network (like V-Net) that processes input images.
  2. The Consistency Branch (Teacher): An ensemble of the student's previous versions that provides stable targets.
  3. The SAM-Assisted Branch: This is the secret sauce.
  4. The student model produces a "coarse" initial prediction.
  5. An uncertainty-aware strategy identifies regions where the student is confident.
  6. These confident regions are converted into "points" used to prompt SAM-Med3D (a version of SAM adapted for 3D medical data).
  7. SAM generates a refined mask, which acts as an additional supervision signal for the student.

The training uses a combined loss function: the student must match the ground truth (for labeled data), stay consistent with the teacher (for unlabeled data), and—crucially—stay consistent with SAM’s refined predictions.

Datasets

The authors evaluated their method on the Left Atrium (LA) Segmentation Challenge Dataset, which contains 100 3D gadolinium-enhanced MRIs. They specifically tested "extremely limited" scenarios using only 1, 2, 4, or 8 labeled cases for training, while treating the rest of the 80 training scans as unlabeled data.

Key Insights and Ideas

  • SAM as a Refiner, Not a Replacement: SAM is famously powerful but often struggles with the specific textures of medical images when used "zero-shot." SemiSAM doesn't rely on SAM to do the whole job; it uses the domain-specific student model to guide SAM to the right spot.
  • Prompting is the Bridge: The student model acts as the "human" in the loop, automatically generating the point prompts that SAM needs to succeed.
  • Uncertainty Filtering: To prevent "garbage in, garbage out," the system only generates prompts from areas where the student model has low uncertainty. This ensures SAM is being asked to refine a likely target rather than hallucinating in a noisy region.
  • Ramping Strategy: The authors used a "ramp-down" weight for the SAM loss. This means SAM is a heavy influencer early in training when the student is lost, but its influence fades as the student becomes a domain expert.

Key Results

SemiSAM showed dramatic improvements over the standard Mean Teacher (MT) baseline and other state-of-the-art methods:

  • With 1 labeled case: The Dice score (a measure of overlap, where 100% is perfect) jumped from 30.60% (MT) to 41.38% (SemiSAM).
  • With 2 labeled cases: Performance rose from 58.17% to 69.46%.
  • With 4 labeled cases: It reached 80.42%, compared to 72.40% for the baseline.

Qualitatively, the 3D reconstructions produced by SemiSAM are much smoother and more anatomically accurate than those produced without the SAM-assisted branch, which often showed fragmented or "noisy" segmentations.

Limitations and Future Work

The authors noted a "ceiling effect": as the number of labeled images increases (e.g., to 10% of the dataset or more), the benefit of using SAM diminishes. This is because a domain-specific model trained on enough data eventually becomes more accurate than a general foundation model like SAM.

Future work includes adapting SemiSAM to the newer SAM 2 architecture and exploring how to make the SAM-assisted branch even more robust for diverse medical imaging tasks beyond the left atrium.

Summary updated

Abstract

Semi-supervised learning has attracted much attention due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods, which is especially important for medical image segmentation which typically requires intensive pixel/voxel-wise labeling by domain experts. Although semi-supervised methods can improve the performance by utilizing unlabeled data, there are still gaps between fully supervised methods under extremely limited annotation scenarios. In this paper, we propose a simple yet efficient strategy to explore the usage of the Segment Anything Model (SAM) for enhancing semi-supervised medical image segmentation. Concretely, the segmentation model trained with domain knowledge provides information for localization and generating input prompts to the SAM. Then the generated pseudo-labels of SAM are utilized as additional supervision to assist in the learning procedure of the semi-supervised framework. Extensive experiments demonstrate that SemiSAM significantly improves the performance of existing semi-supervised frameworks when only one or a few labeled images are available and shows strong efficiency as a plug-and-play strategy for semi-supervised medical image segmentation.
BibTeX
@inproceedings{zhang2024semisam,
  author = {Yichi Zhang and Jin Yang and Yuchen Liu and Yuan Cheng and Yuan Qi},
  title = {{SemiSAM: Enhancing Semi-Supervised Medical Image Segmentation via SAM-Assisted Consistency Regularization}},
  booktitle = {BIBM},
  year = {2024},
  url = {https://openreview.net/forum?id=mxFkEW0rfb},
}

Loading related content...

Figure