Endo-CLIP: Progressive Self-supervised Pre-training on Raw Colonoscopy Records

Yili He, Yan Zhu, Peiyao Fu, Ruijie Yang, Tianyi Chen, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang

Introduction

Self-supervised pre-training on large-scale image-text pairs, exemplified by models like CLIP, has fundamentally shifted the landscape of computer vision. However, translating these successes to the clinical domain—specifically colonoscopy—remains a significant challenge. Raw colonoscopy records are inherently "messy": a single examination typically produces a high volume of video frames (approximately 100 images per exam) paired with a single, unstructured free-text diagnostic report.

This data structure presents three primary obstacles for standard vision-language models:

  1. The "Needle-in-a-Haystack" Problem: The vast majority of frames in a colonoscopy depict healthy bowel tissue, while the diagnostic report focuses exclusively on the rare frames containing polyps.
  2. Complex Terminology: Medical descriptions of polyps involve highly nuanced morphological attributes (size, shape, texture) that require fine-grained semantic understanding.
  3. Multi-Polyp Ambiguity: Approximately 60% of examinations involve multiple polyps. Establishing a precise 1:1 mapping between a specific image and a specific sentence in a report is often impossible without manual intervention.

To address these issues, we introduce Endo-CLIP, a three-stage progressive self-supervised learning framework designed to extract meaningful representations from raw, uncurated colonoscopy data.

Methodology

Endo-CLIP operates through a progressive pipeline that cleanses, aligns, and integrates clinical data to overcome the noise inherent in medical records.

Stage 1: Cleansing (Image Purification)

To solve the "needle-in-a-haystack" problem, the framework first filters out non-informative background frames. Using Large Language Models (LLMs) to parse diagnostic reports for key clinical sentences, the model employs a two-round contrastive learning approach. By calculating the probability that a frame contains a polyp—guided by known prevalence rates—the system discards healthy tissue frames, ensuring the model focuses only on clinically relevant visual data.

Stage 2: Attunement (Single-Polyp Semantic Alignment)

Standard contrastive learning typically uses "hard" one-hot labels, which can be counterproductive in medical imaging where different polyps may share identical visual characteristics. Endo-CLIP utilizes LLMs to extract specific morphological attributes (e.g., sessile vs. pedunculated shapes) to generate "soft" similarity matrices. This allows the model to learn fine-grained clinical features without being penalized for recognizing legitimate similarities between different patients' polyps.

Stage 3: Unification (Multi-Polyp Contextual Integration)

To resolve the ambiguity found in multi-polyp reports, we introduce a polyp-level cross-attention mechanism. Instead of forcing a 1:1 match, each image embedding attends to all textual embeddings within a patient's record, and vice versa. These features are then aggregated at the patient level, allowing the model to holistically align the visual and textual context of an entire examination.

Experimental Setup and Datasets

The model was pre-trained on a private dataset of 13,000 colorectal examination cases, totaling 874,000 endoscopic images and their corresponding reports. To facilitate standardized evaluation, we also introduce EndoReport50, a publicly released benchmark dataset. EndoReport50 contains 84 polyps from 50 patients, featuring detailed textual descriptions, clinician-annotated morphological attributes, and pathology reports. This benchmark is used to evaluate two critical tasks: Polyp Detection and Polyp Malignancy Classification.

Results and Discussion

Endo-CLIP demonstrates significant improvements over supervised baselines (ResNet-50), vision-only models (DINO, SSCD), and standard CLIP.

  • Zero-Shot Performance: In polyp detection, Endo-CLIP achieved a 76.38% AUROC, a +9.93% improvement over standard CLIP. In malignancy classification, the model reached 92.13% AUROC, outperforming standard CLIP by +13.91%.
  • Few-Shot Robustness: The model maintained a dominant lead across 5%, 10%, and 20% few-shot training regimes, suggesting that the pre-trained representations are highly transferable even with minimal labeled data.
  • Feature Representation: t-SNE visualizations indicate that Endo-CLIP creates distinct, well-defined boundaries between benign and malignant polyps. In contrast, other self-supervised models like DINO and SSCD produced overlapping clusters, failing to capture the clinical nuances necessary for malignancy differentiation.

Conclusion

Endo-CLIP demonstrates that noise is the primary enemy of medical vision-language pre-training. By progressively filtering background data, utilizing LLM-guided soft labels, and applying patient-level aggregation, the framework successfully adapts the CLIP paradigm to the complexities of raw clinical records. This work provides a foundation for more accurate, automated endoscopic analysis and real-time clinical decision support.

Summary updated

BibTeX
@inproceedings{he2025endo,
  author = {Yili He and Yan Zhu and Peiyao Fu and Ruijie Yang and Tianyi Chen and Zhihua Wang and Quanlin Li and Pinghong Zhou and Xian Yang and Shuo Wang},
  title = {{Endo-CLIP: Progressive Self-supervised Pre-training on Raw Colonoscopy Records}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2025},
  doi = {10.1007/978-3-032-05141-7\_11},
}
Figure