Abstract
The Segment Anything Model (SAM) has set a new benchmark for general-purpose image segmentation. However, its application in medical imaging, specifically for polyp segmentation in endoscopy, is hindered by domain gaps, overconfidence in out-of-distribution errors, and the requirement for manual prompts. We introduce ASPS (Augmented Segment Anything Model for Polyp Segmentation), a framework that adapts SAM for automated medical use. ASPS incorporates a parallel CNN branch to capture fine-grained local details and an uncertainty-guided regularization strategy to calibrate model confidence. Our approach achieves state-of-the-art performance across five public datasets, demonstrating superior accuracy and generalization without human-in-the-loop prompts.
Introduction
Accurate polyp segmentation is a cornerstone of colorectal cancer diagnosis. While foundation models like SAM excel at segmenting natural objects (e.g., cars or animals), they often struggle with the subtle textures and varied lighting of endoscopic imagery. These "domain gaps" lead to three primary issues: the loss of high-frequency local details, misleadingly high confidence scores for incorrect predictions, and a reliance on manual point or box prompts that are impractical in fast-paced clinical settings.
Methodology
ASPS addresses these challenges through two primary architectural innovations designed to "medicalize" the SAM foundation.
Cross-branch Feature Augmentation (CFA)
While SAM’s Vision Transformer (ViT) backbone is excellent at capturing global context, it often misses the sharp edges and fine textures of polyps that blend into surrounding tissue. ASPS introduces a trainable CNN encoder branch (MSCAN-L) in parallel with the frozen ViT. This branch acts as a specialist for local, high-frequency details. As shown in the architecture overview, these features are fused using a cross-attention mechanism within a modified Mask Decoder, allowing the model to leverage both global shapes and local textures.
Uncertainty-guided Prediction Regularization (UPR)
To mitigate overconfidence in out-of-distribution (OOD) scenarios, ASPS employs a novel regularization strategy. During training, the model generates an uncertainty map ($U_p$) based on its prediction. If the model is uncertain about a specific region, the ground truth (GT) is used as a "hint" to guide the final prediction ($P'$). This mechanism nudges the model toward the correct boundaries during training, effectively calibrating its confidence scores. Additionally, the model is adapted by fine-tuning only the LayerNorm layers in the encoder "neck," preserving the general knowledge of the pre-trained ViT while adapting to medical textures.
Experiments and Results
ASPS was evaluated on five major polyp datasets: Kvasir-SEG and CVC-ClinicDB for training/testing, and CVC-ColonDB, ETIS, and EndoScene for out-of-distribution testing.
- State-of-the-Art Accuracy: On CVC-ClinicDB, ASPS achieved a Dice score of 0.951, surpassing previous leaders like Polyp-PVT (0.937).
- Robust Generalization: The model showed significant improvements on the challenging ETIS dataset, reaching a Dice score of 0.861, compared to 0.750 for SAMUS and 0.687 for MedSAM.
- Frequency Analysis: Fourier transform analysis confirmed that the added CNN branch successfully recovers high-frequency information that the standard ViT branch typically ignores.
Conclusion
ASPS demonstrates that foundation models can be effectively adapted for specialized medical tasks by combining the global strengths of Transformers with the local precision of CNNs. By eliminating the need for manual prompts and introducing uncertainty-guided calibration, ASPS provides a more reliable and automated tool for clinical endoscopy. Future work will focus on optimizing the model for real-time deployment on standard clinical hardware.
Summary updated