TL;DR
This paper introduces SAM-Mamba, a novel architecture that enhances the Segment Anything Model (SAM) for polyp segmentation in colonoscopy images. By integrating a Mamba-based prior module, SAM-Mamba significantly improves zero-shot generalization capabilities, addressing the challenge of segmenting polyps across diverse datasets and acquisition devices with high accuracy.
What Problem Does This Paper Address?
Colorectal cancer (CRC) is a major global health concern, and early detection through polyp segmentation in colonoscopy is crucial. However, segmenting polyps is challenging due to their varied appearance (color, size, shape), indistinct boundaries, and similarity to surrounding tissues. Traditional CNN-based methods struggle with global context, while Vision Transformer (ViT)-based models, including the powerful Segment Anything Model (SAM), often lack domain-specific knowledge for medical imaging, leading to suboptimal performance and poor generalization to unseen data. Fine-tuning SAM is computationally expensive and can lead to overfitting. This paper aims to bridge this gap by developing an efficient and highly generalizable polyp segmentation method that leverages SAM's strengths while incorporating domain-specific features.
Methods
The core innovation of SAM-Mamba is the integration of a "Mamba-Prior" module into the SAM architecture. This module is designed to inject polyp-relevant, multi-scale, and global contextual information into SAM's image encoder, enhancing its domain-specific understanding. The Mamba-Prior module comprises three key strategies:
- Multi-scale Spatial Decomposition (MSD): This process uses parallel convolutional layers with different receptive fields (3x3, 5x5, 7x7) to capture low-level features (size, shape, boundary) of polyps at various spatial scales. This helps in analyzing fine-grained to coarse semantics.
- Channel Saliency and Context Accumulation: For each scale of spatial decomposition, salient and contextual cues are extracted and accumulated along the channel depth.
- Mamba Channel Interaction: Leveraging Mamba's ability to model long-range dependencies efficiently, this component captures intra-channel variations in the extracted salient and contextual features across multiple spatial scales. This helps in understanding complex polyp regions and their boundaries.
Instead of full fine-tuning, SAM-Mamba employs an "Adapter-based" fine-tuning approach. Adapters, which are small bottleneck modules, are trained to adapt SAM to the polyp segmentation task. The Mamba-Prior module is learned alongside these adapters to inject critical polyp domain-specific cues into the SAM encoder. The model is trained in two stages: first, the adapters are trained with deep supervision from the encoder's output, and then the entire model (including the SAM decoder) is trained with full supervision.
Datasets
The paper evaluates SAM-Mamba on five benchmark polyp segmentation datasets:
- Seen Datasets: Kvasir-SEG and CVC-ClinicDB.
- Unseen Datasets (for zero-shot evaluation): CVC-300, CVC-ColonDB, and ETIS.
The training set consists of 1,450 images (900 from Kvasir-SEG, 550 from CVC-ClinicDB), and the testing set includes images from all five datasets, with specific splits for Kvasir-SEG and CVC-ClinicDB for testing. This configuration is designed to challenge the model with varying resolutions and acquisition devices.
Key Insights and Ideas
- Mamba for Medical Imaging: The paper demonstrates the effectiveness of Mamba, a recent state-space model architecture, in capturing long-range dependencies crucial for medical image segmentation, particularly for complex structures like polyps.
- Bridging General and Specific: SAM-Mamba successfully bridges the gap between SAM's general segmentation capabilities and the specific requirements of polyp segmentation by injecting domain-specific priors via the Mamba-Prior module.
- Efficient Adaptation: The adapter-based fine-tuning strategy offers a computationally efficient way to adapt large pre-trained models like SAM to specialized tasks, mitigating issues like overfitting and high computational costs.
- Zero-Shot Generalization is Key: The emphasis on evaluating performance on unseen datasets highlights the critical need for models that can generalize well in real-world clinical settings, where data variability is high.
Key Results
SAM-Mamba demonstrates superior performance across multiple benchmark datasets:
- Seen Datasets: On Kvasir-SEG, SAM-Mamba achieves a mDice of 92.4%, outperforming competitive methods like CTNet. On CVC-ClinicDB, it also shows strong results, outperforming peers in key metrics.
- Unseen Datasets (Zero-Shot): SAM-Mamba significantly outperforms SOTA models on unseen datasets. For instance, on CVC-ColonDB, it achieves a +4% improvement in mIoU, and on ETIS, it shows a +3.8% improvement in mIoU compared to other methods.
- Qualitative Results: Visual comparisons show SAM-Mamba's ability to accurately segment polyps of diverse sizes and textures, with a notably low false positive rate, even on unseen datasets. The Mamba-Prior module is credited with reducing false positives by effectively exploring global context.
Limitations and Future Work
- Edge Detection: While SAM-Mamba excels in overall segmentation, the paper notes that scores for Emax (edge detection) indicate a slight room for improvement.
- Future Work: The paper suggests that the multi-scale approach, particularly with the Mamba-Prior module, enhances generalization across different data scales, indicating potential for further exploration in this direction. The authors also implicitly suggest that further refinement of the adapter and Mamba components could lead to even better performance.
Summary updated