TL;DR
SemiSAM-O1 is an annotation-efficient semi-supervised medical image segmentation framework that requires only a single annotated template image to train a high-performing model. By leveraging a generalist foundation model offline as a feature extractor and employing an uncertainty-guided KNN refinement loop, it dramatically cuts down training time while significantly narrowing the performance gap with fully supervised models.
What Problem Does This Paper Address?
Deep learning models for medical image segmentation typically require massive amounts of expert-annotated data, which is incredibly expensive and labor-intensive to acquire. While semi-supervised learning (SSL) attempts to mitigate this by leveraging unlabeled data, standard methods suffer from severe performance degradation in extreme low-data regimes—such as having only one labeled case.
Recent foundation-model-driven approaches (like SAM-based frameworks) try to bridge this gap by enforcing consistency between a specialist model and a generalist foundation model. However, as shown in the figure below, these methods repeatedly call the massive foundation model online during training, incurring massive computational overhead. Furthermore, they lack quality control over noisy or incorrect foundation model predictions, which can inadvertently corrupt the training process.
Methods
SemiSAM-O1 introduces a two-stage framework that completely decouples the foundation model from the active training loop, unlocking its feature representation capabilities beyond a simple prompting interface:
- Offline Feature Extraction & Prototype Propagation (Round 0): A frozen foundation model encoder (such as SAM-Med3D) extracts spatial and global features for all volumes once before training begins. Class prototypes derived from the single annotated template are propagated to the unlabeled pool via feature similarity to produce coarse initial pseudo-labels.
- Iterative Training-and-Refinement Loop (Rounds 1 to R): The pipeline enters a multi-round loop. In each round, a specialist model (e.g., 3D U-Net) is trained from scratch using the current pseudo-labels and the single real label. The trained specialist then generates updated pseudo-labels alongside voxel-wise uncertainty estimates (entropy). An uncertainty-guided K-Nearest Neighbor (KNN) refinement step identifies high-uncertainty regions and corrects them by aggregating labels from their most similar, confident neighbors in the foundation model's global feature space.
Datasets
The authors comprehensively evaluated SemiSAM-O1 across four 3D datasets covering diverse modalities and anatomical targets under an extreme one-label setting ($M=1$):
- Left Atrium (LA) Challenge Dataset: 100 3D gadolinium-enhanced MRIs.
- BraTS 2019: 250 FLAIR MRI scans for brain tumor segmentation (highly heterogeneous targets).
- PETS (In-house): 100 FDG-PET cardiac images, representing a completely unseen functional imaging modality for the pre-trained foundation model.
- RT-EC (In-house): 84 CT scans from endometrial cancer patients for clinical target volume (CTV) and planning target volume (PTV) segmentation.
Key Insights and Ideas
- Unlocking the Feature Space: Instead of treating foundation models merely as interactive, prompt-based segmentors, SemiSAM-O1 treats them as global semantic embedding generators. This allows the framework to exploit rich, pre-trained representations offline.
- Mitigating Confirmation Bias: Pseudo-labeling loops often suffer from confirmation bias, where a model repeatedly reinforces its own mistakes. Re-initializing the specialist model at each round and using the foundation model's global feature space for KNN refinement successfully breaks this error-propagation loop.
- Decoupled Compute Efficiency: Shifting the foundation model's role to a one-time offline feature extractor reduces generalist model inference time to nearly zero during training, making it substantially faster than online prompting methods.
Key Results
- Performance Boost: On the Left Atrium dataset, SemiSAM-O1 boosted the Dice score from 41.73% (standard SSL baseline) and 50.10% (SemiSAM+) to 76.78% by the second iteration round, drastically closing the gap to the fully supervised upper bound (88.18%).
- Robustness to Unseen Modalities: On the PETS dataset (functional PET data, which SAM-Med3D never encountered during pre-training), SemiSAM-O1 achieved 68.40% Dice, vastly outperforming SemiSAM+ (34.24%), which suffered from generalization failure during online prompting.
- Training Speedup: SemiSAM-O1 required only 10.4 hours of training compared to SemiSAM+'s 15.7 hours, representing a 34% reduction in training time despite running multiple iterative rounds from scratch.
Limitations and Future Work
- Foundation Model Dependency: The quality of the initial prototype propagation still hinges on the underlying foundation model's representation capability. If the domain gap is too massive, the initial pseudo-labels may be too weak to recover from.
- Structural Consistency Assumption: The KNN refinement step assumes that global semantic similarity implies voxel-level structural alignment. This assumption degrades in highly irregular targets like brain tumors (BraTS dataset), where tumor shape, size, and location vary wildly across patients.
- Future Directions: The authors suggest exploring multi-foundation model pools, agentic frameworks to dynamically select or combine models, and incorporating deformable feature alignment or local feature matching to handle complex structural variability.
Summary updated