Unlocking Generalization in Polyp Segmentation with DINO Self-Attention "keys"

Carla Monteiro, Valentina Corbetta, Regina Beets-Tan, Luís F. Teixeira, Wilson SilvaarXiv 2025 · DOI · arXiv

TL;DR

This paper tackles the challenge of polyp segmentation in colonoscopy images, where existing deep learning models struggle with generalization to unseen data. The authors propose a novel approach that leverages the robust "key" features from DINO self-supervised learning, combined with a simple convolutional decoder, to achieve state-of-the-art generalization performance without relying on complex, task-specific architectures.

What Problem Does This Paper Address?

Accurate polyp segmentation in colonoscopy images is crucial for early detection of colorectal cancer. However, current deep learning models often fail to generalize well to new datasets or variations in imaging conditions (e.g., lighting, equipment). This is partly due to models learning spurious correlations from the training data rather than focusing on clinically relevant features. Existing methods often require complex, tailored architectures, making them difficult to adapt to new scenarios. The paper aims to develop a more robust and generalizable polyp segmentation method that overcomes these limitations.

Methods

The core of the proposed method lies in utilizing the "key" features extracted from the self-attention mechanism of Vision Transformers (ViTs) pre-trained with DINO (Self-Distillation With No Labels). Unlike traditional approaches that use tokens from the deepest layers, this work focuses on the "key" features, which are shown to be more object-centered and robust. These features are then fed into a simple, lightweight CNN-based decoder to predict the polyp masks. The authors systematically evaluate different versions of DINO (DINOv1, DINOv2, DINOv3) and their architectural advancements, demonstrating how these contribute to improved segmentation performance. They also conduct ablation studies to confirm that task-specific fine-tuning of the DINO features is not necessary, highlighting the inherent robustness of the self-supervised representations.

Datasets

The primary dataset used for evaluation is PolypGen, a multi-center colonoscopy dataset containing 3,762 annotated polyp images. The analysis focuses on the single-frame subset, comprising 1,537 samples from six different institutions. The paper also leverages datasets used for DINO pre-training: ImageNet for DINOv1, LVD-142M for DINOv2, and LVD-1689M for DINOv3.

Key Insights and Ideas

  • DINO "key" features are inherently robust: The paper strongly suggests that the "key" features learned through DINO self-supervised pre-training are highly effective for polyp segmentation, capturing essential visual information without explicit task-specific supervision.
  • Simplicity can be powerful: A simple convolutional decoder, when paired with powerful pre-trained features, can achieve state-of-the-art results, challenging the need for complex, task-specific architectures.
  • DINO evolution matters: The study demonstrates a clear progression in performance with newer DINO versions (especially DINOv2 with register tokens and DINOv3), highlighting the impact of architectural advancements in self-supervised learning.
  • Generalization is unlocked: The proposed method shows significantly improved generalization capabilities, particularly in challenging, data-scarce scenarios, making it more suitable for real-world clinical deployment.

Key Results

The proposed method, particularly using DINOv3, achieved state-of-the-art (SOTA) performance on both Domain Generalization (DG) and Extreme Single Domain Generalization (ESDG) tests.

  • DG Test: The proposed model achieved a DSC of 80.00% ± 8.31, outperforming baselines like nnU-Net (65.09% ± 14.75) and UM-Netloss (73.26% ± 13.42).
  • ESDG Test: The proposed model achieved a DSC of 73.64% ± 3.33, again surpassing nnU-Net (52.36% ± 7.97) and UM-Netloss (63.22% ± 3.23).
  • The performance drop from DG to ESDG was significantly smaller for the proposed model (8-10%) compared to baselines like U-Net (around 37%).
  • Statistical analysis confirmed these performance differences were statistically significant.
  • The proposed variants also demonstrated competitive or superior computational efficiency compared to some baselines.

Limitations and Future Work

  • Reliance on pre-trained weights: The authors acknowledge that using publicly available DINO weights meant they couldn't always ensure identical pre-training conditions (e.g., patch size) across all versions, which might have slightly impacted comparisons.
  • ViT-s architecture: For computational reasons, the ViT-small architecture was used as the feature extractor. Using larger ViT variants could potentially lead to even greater performance gains.
  • Future Work: The authors plan to extend this approach to polyp video segmentation, investigate its transferability to other segmentation tasks (e.g., tumors in CT/MRI), and explore the use of larger ViT versions for further performance enhancement. They also note that their proposed variants meet the inference time requirements for real-time video integration.

Summary updated

Abstract

Automatic polyp segmentation is crucial for improving the clinical identification of colorectal cancer (CRC). While Deep Learning (DL) techniques have been extensively researched for this problem, current methods frequently struggle with generalization, particularly in data-constrained or challenging settings. Moreover, many existing polyp segmentation methods rely on complex, task-specific architectures. To address these limitations, we present a framework that leverages the intrinsic robustness of DINO self-attention "key" features for robust segmentation. Unlike traditional methods that extract tokens from the deepest layers of the Vision Transformer (ViT), our approach leverages the key features of the self-attention module with a simple convolutional decoder to predict polyp masks, resulting in enhanced performance and better generalizability. We validate our approach using a multi-center dataset under two rigorous protocols: Domain Generalization (DG) and Extreme Single Domain Generalization (ESDG). Our results, supported by a comprehensive statistical analysis, demonstrate that this pipeline achieves state-of-the-art (SOTA) performance, significantly enhancing generalization, particularly in data-scarce and challenging scenarios. While avoiding a polyp-specific architecture, we surpass well-established models like nnU-Net and UM-Net. Additionally, we provide a systematic benchmark of the DINO framework's evolution, quantifying the specific impact of architectural advancements on downstream polyp segmentation performance.
BibTeX
@misc{monteiro2025unlocking,
  author = {Carla Monteiro and Valentina Corbetta and Regina Beets-Tan and Luís F. Teixeira and Wilson Silva},
  title = {{Unlocking Generalization in Polyp Segmentation with DINO Self-Attention "keys"}},
  year = {2025},
  eprint = {2512.13376},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.13376},
}
Figure