SemiSAM-O1: Pushing the Boundary of Annotation-Efficient Medical Image Segmentation with Generalist Knowledge Fusion

Yichi Zhang, Le Xue, Bichun Xu, Judong Luo, Zhigang Wu, Yu Fu, Zixin Hu, Yuan Cheng, Yuan QiarXiv 2026 · arXiv

TL;DR

SemiSAM-O1 is an annotation-efficient semi-supervised medical image segmentation framework that requires only a single annotated template image to train a high-performing model. By leveraging a generalist foundation model offline as a feature extractor and employing an uncertainty-guided KNN refinement loop, it dramatically cuts down training time while significantly narrowing the performance gap with fully supervised models.

What Problem Does This Paper Address?

Deep learning models for medical image segmentation typically require massive amounts of expert-annotated data, which is incredibly expensive and labor-intensive to acquire. While semi-supervised learning (SSL) attempts to mitigate this by leveraging unlabeled data, standard methods suffer from severe performance degradation in extreme low-data regimes—such as having only one labeled case.

Recent foundation-model-driven approaches (like SAM-based frameworks) try to bridge this gap by enforcing consistency between a specialist model and a generalist foundation model. However, as shown in the figure below, these methods repeatedly call the massive foundation model online during training, incurring massive computational overhead. Furthermore, they lack quality control over noisy or incorrect foundation model predictions, which can inadvertently corrupt the training process.

Fig. 1. Comparison of SemiSAM+ and the proposed SemiSAM-O1. SemiSAM+ repeatedly invokes SAM inference during training to compute a consistency loss between the specialist model and SAM outputs, incurring substantial computational overhead with no quality control on SAM’s output reliability. In contrast, SemiSAM-O1 employs SAM only once in an offline stage to learn representations for prototype-based pseudo-label initialization, after which all iterative rounds of specialist training, pseudo-label generation and refinement proceed without SAM involvement.

Methods

SemiSAM-O1 introduces a two-stage framework that completely decouples the foundation model from the active training loop, unlocking its feature representation capabilities beyond a simple prompting interface:

  1. Offline Feature Extraction & Prototype Propagation (Round 0): A frozen foundation model encoder (such as SAM-Med3D) extracts spatial and global features for all volumes once before training begins. Class prototypes derived from the single annotated template are propagated to the unlabeled pool via feature similarity to produce coarse initial pseudo-labels.
  2. Iterative Training-and-Refinement Loop (Rounds 1 to R): The pipeline enters a multi-round loop. In each round, a specialist model (e.g., 3D U-Net) is trained from scratch using the current pseudo-labels and the single real label. The trained specialist then generates updated pseudo-labels alongside voxel-wise uncertainty estimates (entropy). An uncertainty-guided K-Nearest Neighbor (KNN) refinement step identifies high-uncertainty regions and corrects them by aggregating labels from their most similar, confident neighbors in the foundation model's global feature space.
Fig. 2. Overview of the proposed SemiSAM-O1 framework. (a) The generalist foundation model encoder extracts spatial and global features from all volumes once before training. (b) In Round 0, class prototypes computed from a single labeled volume are propagated to the unlabeled pool via cosine similarity to produce initial pseudo-labels. (c) In each subsequent round, a specialist SSL model is trained under conventional semi-supervised supervision on the real label and generalist model-based supervision on pseudo-labels that are updated between rounds. (d) After training, the specialist generates new pseudo-labels with uncertainty estimates, and an uncertainty-guided KNN refinement step corrects low-confidence predictions by querying their nearest neighbors in the foundation model’s feature space. Components (c) and (d) repeat for R rounds to progressively improve pseudo-label quality and model performance.

Datasets

The authors comprehensively evaluated SemiSAM-O1 across four 3D datasets covering diverse modalities and anatomical targets under an extreme one-label setting ($M=1$):

  • Left Atrium (LA) Challenge Dataset: 100 3D gadolinium-enhanced MRIs.
  • BraTS 2019: 250 FLAIR MRI scans for brain tumor segmentation (highly heterogeneous targets).
  • PETS (In-house): 100 FDG-PET cardiac images, representing a completely unseen functional imaging modality for the pre-trained foundation model.
  • RT-EC (In-house): 84 CT scans from endometrial cancer patients for clinical target volume (CTV) and planning target volume (PTV) segmentation.

Key Insights and Ideas

  • Unlocking the Feature Space: Instead of treating foundation models merely as interactive, prompt-based segmentors, SemiSAM-O1 treats them as global semantic embedding generators. This allows the framework to exploit rich, pre-trained representations offline.
  • Mitigating Confirmation Bias: Pseudo-labeling loops often suffer from confirmation bias, where a model repeatedly reinforces its own mistakes. Re-initializing the specialist model at each round and using the foundation model's global feature space for KNN refinement successfully breaks this error-propagation loop.
  • Decoupled Compute Efficiency: Shifting the foundation model's role to a one-time offline feature extractor reduces generalist model inference time to nearly zero during training, making it substantially faster than online prompting methods.

Key Results

  • Performance Boost: On the Left Atrium dataset, SemiSAM-O1 boosted the Dice score from 41.73% (standard SSL baseline) and 50.10% (SemiSAM+) to 76.78% by the second iteration round, drastically closing the gap to the fully supervised upper bound (88.18%).
  • Robustness to Unseen Modalities: On the PETS dataset (functional PET data, which SAM-Med3D never encountered during pre-training), SemiSAM-O1 achieved 68.40% Dice, vastly outperforming SemiSAM+ (34.24%), which suffered from generalization failure during online prompting.
  • Training Speedup: SemiSAM-O1 required only 10.4 hours of training compared to SemiSAM+'s 15.7 hours, representing a 34% reduction in training time despite running multiple iterative rounds from scratch.
Fig. 4. Effect of the number of iterative refinement rounds of SemiSAM-O1 on segmentation performance (Dice) using the MT and UA-MT backbones on the LA dataset.

Limitations and Future Work

  • Foundation Model Dependency: The quality of the initial prototype propagation still hinges on the underlying foundation model's representation capability. If the domain gap is too massive, the initial pseudo-labels may be too weak to recover from.
  • Structural Consistency Assumption: The KNN refinement step assumes that global semantic similarity implies voxel-level structural alignment. This assumption degrades in highly irregular targets like brain tumors (BraTS dataset), where tumor shape, size, and location vary wildly across patients.
  • Future Directions: The authors suggest exploring multi-foundation model pools, agentic frameworks to dynamically select or combine models, and incorporating deformable feature alignment or local feature matching to handle complex structural variability.

Summary updated

Abstract

Semi-supervised learning (SSL) has become a promising solution to alleviate the annotation burden of deep learning-based medical image segmentation models. While recent advances in foundation model-driven SSL have pushed the boundary to extremely limited annotation scenarios, they fail to maintain robust competitive performance in complex imaging modalities. In this paper, we propose SemiSAM-O1, an annotation-efficient framework using only one annotated template case for segmentation. SemiSAM-O1 extends the specialist-generalist collaborative learning framework to the extreme one-label setting by fully exploiting the foundation model's feature representation capability beyond its prompting interface. SemiSAM-O1 operates in two stages. In the first stage, the foundation model's encoder extracts dense features from all volumes, and class prototypes derived from the single annotated template are propagated to the unlabeled pool via feature similarity to produce coarse initial pseudo-labels. In the second stage, an iterative training-and-refinement loop progressively improves both the segmentation model and the pseudo-labels over multiple rounds, where each round trains the model from scratch on current pseudo-labels and generates updated predictions with voxel-wise uncertainty estimates. An uncertainty-guided refinement step further leverages the foundation model's global feature space to correct high-uncertainty regions by aggregating labels from their most similar confident neighbors, establishing a virtuous cycle of mutual improvement. Extensive experiments on a wide range of segmentation tasks across different modalities and anatomical targets demonstrate that SemiSAM-O1 significantly narrows the performance gap between one-label semi-supervised learning and full supervision, while significantly reducing the computational overhead of online foundation model inference.
BibTeX
@misc{zhang2026semisam,
  author = {Yichi Zhang and Le Xue and Bichun Xu and Judong Luo and Zhigang Wu and Yu Fu and Zixin Hu and Yuan Cheng and Yuan Qi},
  title = {{SemiSAM-O1: Pushing the Boundary of Annotation-Efficient Medical Image Segmentation with Generalist Knowledge Fusion}},
  year = {2026},
  eprint = {2604.24109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.24109},
}
Figure