Introduction
Colorectal cancer prevention relies heavily on the accurate segmentation of polyps during colonoscopies. While deep learning has significantly advanced this field, most state-of-the-art models are "data-hungry," requiring thousands of pixel-wise manual annotations. For medical experts, delineating every polyp pixel-by-pixel is an exhausting, time-consuming, and costly endeavor.
Weak supervision—using "scribbles" (simple lines drawn over the foreground and background)—offers a much faster alternative, but often at the cost of poor boundary definition. Meanwhile, general-purpose foundation models like the Segment Anything Model (SAM) excel at broad segmentation tasks but frequently struggle in specialized medical contexts due to a lack of domain-specific knowledge. This paper introduces the SAM-guided Collaborative Learning Network (SAM-CLNet), a framework designed to bridge the gap between low-cost weak supervision and high-performance segmentation by effectively "teaching" SAM about medical images while using its outputs to guide a specialized network.
Methodology
The SAM-CLNet framework is built upon two primary pillars that interact through an online collaborative learning loop:
1. CEA-Net (Cross-level Enhancement and Aggregation Network)
CEA-Net serves as the primary segmentation model, designed to handle the inherent variability in polyp size and appearance. It utilizes two novel modules:
- Cross-level Enhancement Module (CEM): This module fuses features from adjacent encoder layers, ensuring that high-resolution spatial details and low-resolution semantic information are integrated effectively.
- Feature Aggregation Module (FAM): It cascades multi-level features to capture a richer representation of the polyp, improving robustness against variations in color and scale.
2. SAM-guided Mask Generation (SAM-MGNet)
SAM-MGNet acts as a "teacher" by providing high-quality pseudo-labels. To optimize SAM's performance, the authors developed a Box-Augmentation Strategy. Instead of relying solely on a scribble, the model combines the scribble with its current prediction to create a "prompt box." By intersecting an expanded scribble box with the predicted mask box, the system generates a balanced, accurate prompt for SAM.
To ensure quality, the framework includes an Image-level Filtering mechanism. If the similarity between SAM’s output and the human scribbles falls below a threshold ($\tau < 0.5$), the mask is discarded to prevent the model from learning from incorrect or "hallucinated" labels.
Experimental Setup and Datasets
The study utilized five benchmark colonoscopy datasets. The authors re-labeled 1,450 images with scribble annotations—a process taking only 3–4 seconds per image—and released this dataset to the community. Training and testing were conducted on Kvasir and CVC-ClinicDB, while CVC-300, CVC-ColonDB, and ETIS-LaribPolypDB were used to evaluate the model's generalization capabilities on unseen data.
Results and Discussion
SAM-CLNet consistently outperformed existing state-of-the-art weakly-supervised methods across all benchmarks. On the Kvasir dataset, it achieved a mean Dice (mDice) of 0.857, notably outperforming the base CEA-Net (0.842) and other competitive models like C2FNet (0.819).
Key insights from the results include:
- Generalization: On the challenging ETIS-LaribPolypDB dataset, SAM-CLNet showed a 6.4% improvement in mDice over the base model, demonstrating superior performance on unseen data.
- Online Collaborative Learning: Unlike offline methods that generate labels once, the online approach allows the model to refine its boundaries as the "student" network improves, leading to significantly better handling of indistinct edges and small polyps.
- Prompt Strategy: The intersection of expanded scribble boxes and predicted masks proved critical in preventing prompts from being too restrictive or overly broad.
Conclusion
SAM-CLNet demonstrates that foundation models like SAM can be successfully adapted for specialized medical tasks through collaborative learning and intelligent prompting. By narrowing the performance gap between scribble-based supervision and full manual annotation, this framework offers a scalable solution for high-accuracy medical image segmentation. Future work may extend this collaborative framework to other modalities, such as skin lesion segmentation or COVID-19 lung infection detection in CT scans.
Summary updated