Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation

Yiming Zhao, Tao Zhou, Yunqi Gu, Yi Zhou, Yizhe Zhang, Ye Wu, Huazhu FuarXiv 2023 · DOI · arXiv

Introduction

Colorectal cancer prevention relies heavily on the accurate segmentation of polyps during colonoscopies. While deep learning has significantly advanced this field, most state-of-the-art models are "data-hungry," requiring thousands of pixel-wise manual annotations. For medical experts, delineating every polyp pixel-by-pixel is an exhausting, time-consuming, and costly endeavor.

Weak supervision—using "scribbles" (simple lines drawn over the foreground and background)—offers a much faster alternative, but often at the cost of poor boundary definition. Meanwhile, general-purpose foundation models like the Segment Anything Model (SAM) excel at broad segmentation tasks but frequently struggle in specialized medical contexts due to a lack of domain-specific knowledge. This paper introduces the SAM-guided Collaborative Learning Network (SAM-CLNet), a framework designed to bridge the gap between low-cost weak supervision and high-performance segmentation by effectively "teaching" SAM about medical images while using its outputs to guide a specialized network.

Methodology

The SAM-CLNet framework is built upon two primary pillars that interact through an online collaborative learning loop:

1. CEA-Net (Cross-level Enhancement and Aggregation Network)

CEA-Net serves as the primary segmentation model, designed to handle the inherent variability in polyp size and appearance. It utilizes two novel modules:

  • Cross-level Enhancement Module (CEM): This module fuses features from adjacent encoder layers, ensuring that high-resolution spatial details and low-resolution semantic information are integrated effectively.
  • Feature Aggregation Module (FAM): It cascades multi-level features to capture a richer representation of the polyp, improving robustness against variations in color and scale.

2. SAM-guided Mask Generation (SAM-MGNet)

SAM-MGNet acts as a "teacher" by providing high-quality pseudo-labels. To optimize SAM's performance, the authors developed a Box-Augmentation Strategy. Instead of relying solely on a scribble, the model combines the scribble with its current prediction to create a "prompt box." By intersecting an expanded scribble box with the predicted mask box, the system generates a balanced, accurate prompt for SAM.

Overview of the SAM-CLNet architecture, highlighting the collaborative learning loop between CEA-Net and the SAM-guided Mask Generation module.

To ensure quality, the framework includes an Image-level Filtering mechanism. If the similarity between SAM’s output and the human scribbles falls below a threshold ($\tau < 0.5$), the mask is discarded to prevent the model from learning from incorrect or "hallucinated" labels.

Experimental Setup and Datasets

The study utilized five benchmark colonoscopy datasets. The authors re-labeled 1,450 images with scribble annotations—a process taking only 3–4 seconds per image—and released this dataset to the community. Training and testing were conducted on Kvasir and CVC-ClinicDB, while CVC-300, CVC-ColonDB, and ETIS-LaribPolypDB were used to evaluate the model's generalization capabilities on unseen data.

Results and Discussion

SAM-CLNet consistently outperformed existing state-of-the-art weakly-supervised methods across all benchmarks. On the Kvasir dataset, it achieved a mean Dice (mDice) of 0.857, notably outperforming the base CEA-Net (0.842) and other competitive models like C2FNet (0.819).

Quantitative performance comparison of SAM-CLNet against state-of-the-art fully-supervised and weakly-supervised methods on the Kvasir and CVC-ClinicDB datasets.

Key insights from the results include:

  • Generalization: On the challenging ETIS-LaribPolypDB dataset, SAM-CLNet showed a 6.4% improvement in mDice over the base model, demonstrating superior performance on unseen data.
  • Online Collaborative Learning: Unlike offline methods that generate labels once, the online approach allows the model to refine its boundaries as the "student" network improves, leading to significantly better handling of indistinct edges and small polyps.
  • Prompt Strategy: The intersection of expanded scribble boxes and predicted masks proved critical in preventing prompts from being too restrictive or overly broad.

Conclusion

SAM-CLNet demonstrates that foundation models like SAM can be successfully adapted for specialized medical tasks through collaborative learning and intelligent prompting. By narrowing the performance gap between scribble-based supervision and full manual annotation, this framework offers a scalable solution for high-accuracy medical image segmentation. Future work may extend this collaborative framework to other modalities, such as skin lesion segmentation or COVID-19 lung infection detection in CT scans.

Summary updated

Abstract

Polyp segmentation plays a vital role in accurately locating polyps at an early stage, which holds significant clinical importance for the prevention of colorectal cancer. Various polyp segmentation methods have been developed using fully-supervised deep learning techniques. However, pixel-wise annotation for polyp images by physicians during the diagnosis is both time-consuming and expensive. Moreover, visual foundation models such as the Segment Anything Model (SAM) have shown remarkable performance. Nevertheless, directly applying SAM to medical segmentation may not produce satisfactory results due to the inherent absence of medical knowledge. In this paper, we propose a novel SAM-guided Collaborative Learning Network (SAM-CLNet) for scribble-supervised polyp segmentation, enabling a collaborative learning process between our segmentation network and SAM to boost the model performance. Specifically, we first propose a Cross-level Enhancement and Aggregation Network (CEA-Net) for weakly-supervised polyp segmentation. Within CEA-Net, we propose a Cross-level Enhancement Module (CEM) that integrates the adjacent features to enhance the representation capabilities of different resolution features. Additionally, a Feature Aggregation Module (FAM) is employed to capture richer features across multiple levels. Moreover, we present a box-augmentation strategy that combines the segmentation maps generated by CEA-Net with scribble annotations to create more precise prompts. These prompts are then fed into SAM, generating segmentation SAM-guided masks, which can provide additional supervision to train CEA-Net effectively. Furthermore, we present an Image-level Filtering Mechanism to filter out unreliable SAM-guided masks. Extensive experimental results show that our SAM-CLNet outperforms state-of-the-art weakly-supervised segmentation methods.
BibTeX
@misc{zhao2023segment,
  author = {Yiming Zhao and Tao Zhou and Yunqi Gu and Yi Zhou and Yizhe Zhang and Ye Wu and Huazhu Fu},
  title = {{Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation}},
  year = {2023},
  eprint = {2312.00312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.00312},
}
Figure