Attention Residual U-Net for Enhanced Cross-Dataset Polyp Localization in Endoscopy

Romil Imtiaz, Dimitris K. IakovidisarXiv 2026 · arXiv

TL;DR

This paper introduces a novel Attention Residual U-Net designed to solve the "domain shift" problem in endoscopic polyp localization. By replacing direct bounding-box regression with lesion-focused spatial representations and using attention gates to filter out background noise, the framework achieves up to a 22% improvement in AUC over state-of-the-art YOLO models when tested across different clinical datasets.

What Problem Does This Paper Address?

Colorectal polyps are precancerous lesions; missing them during an endoscopy can lead to late-stage cancer. While deep learning models like YOLO have become the standard for automated detection, they often fail in "real-world" clinical settings.

The core challenge is cross-dataset variability. A model trained on one hospital's data often struggles when faced with a different camera brand, varying lighting conditions, or artifacts like mucosal folds and specular reflections. Standard detectors tend to propagate this "background noise" through the network, leading to false positives or missed detections of small, low-contrast polyps. The authors aim to build a system that is robust enough to maintain high performance even when the testing data looks significantly different from the training data.

Methods

The authors move away from the standard "black box" bounding-box regression. Instead, they propose a detection-oriented localization framework based on a modified U-Net architecture.

Fig. 1. Overall architecture of the proposed detection-oriented Attention Residual U-Net with residual encoding, attention-guided feature refinement, and multi-scale contextual learning.

The framework relies on three technical pillars:

  1. Residual Learning: The encoder uses residual blocks to improve gradient flow and allow for more stable training of deeper architectures, which is essential for capturing complex textures.
  2. Attention-Guided Feature Suppression: Instead of passing all encoder data directly to the decoder via skip connections, the model uses Attention Gates. These gates use features from the decoder to "guide" the encoder, effectively filtering out irrelevant background activations (like reflections or tissue folds) before they can interfere with localization.
  3. Multi-Scale Contextual Representation: A bottleneck module sits between the encoder and decoder, using parallel dilated convolutions with different dilation rates. This allows the network to "see" at multiple scales simultaneously—crucial for detecting both tiny early-stage polyps and large lesions.

Finally, the model outputs a spatial confidence map, which is converted into a standard bounding box via post-processing for compatibility with existing medical evaluation protocols.

Datasets

The study emphasizes cross-dataset robustness, using one large source for training and three distinct datasets for external testing:

  • LDPolyp (Source): A large-scale benchmark used for training and internal validation (33,884 polyp images).
  • KUMC (External): 103 polyp images used for cross-dataset evaluation.
  • Kvasir-SEG (External): 1,000 images representing a different clinical distribution.
  • EDD2020 (External): 127 images from an endoscopy disease detection challenge.

Key Insights and Ideas

  • Spatial Maps > Direct Regression: By learning a spatial "heat map" of the lesion first, the model retains fine-grained details that are often lost when a network tries to jump straight to predicting four coordinates for a bounding box.
  • The "Anti-Noise" Filter: The attention mechanism acts as a gatekeeper. In endoscopy, a bright reflection on a wet tissue surface can look a lot like a lesion to a standard CNN. The attention gates suppress these "unstable" activations by comparing them against the broader context known by the decoder.
  • Scale Matters: Polyps don't come in one size. The use of dilated convolutions in the bottleneck ensures that the model doesn't have to choose between "local texture" and "global context"—it gets both.

Key Results

The proposed framework consistently outperformed YOLOv7 and YOLOv11, particularly in cross-dataset scenarios where domain shift is most prevalent.

  • LDPolyp (Internal): Achieved a near-perfect 97.0% AUC and 98.6% F1-score, significantly higher than YOLOv7 (79.5% AUC).
  • KUMC (Cross-Dataset): While all models saw a performance drop due to domain shift, the proposed model maintained a 76.5% AUC, outperforming YOLOv11 by 8.5% absolute points.
  • Kvasir-SEG: The model showed exceptional robustness here, reaching a 95.0% AUC compared to YOLOv7’s 52.1%.

Qualitatively, the heatmaps generated by the Attention Residual U-Net are much more "compact" and centered on the actual lesion, whereas YOLO models often produced diffuse activations that bled into the surrounding healthy tissue.

Fig. 5. Qualitative comparison of activation responses and localization outputs for YOLOv7, YOLOv11, and the proposed framework under challenging cross-dataset conditions. rom top to bottom, the corresponding rows show representative samples from the KUMC, EDD2020, Kvasir-SEG, and LDPolyp datasets, respectively. (a) Ground truth lesion annotations. (b) YOLOv7 outputs. (c) YOLOv11 outputs. (d). Green boxes indicate ground truth lesion regions and red boxes indicate predicted localization outputs. The heatmaps illustrate the spatial activation behavior of each method during lesion localization.

Limitations and Future Work

The authors acknowledge that while their model is more robust, sensitivity still drops under "severe domain variation" (as seen in the KUMC results). When faced with completely unfamiliar visual patterns, the model tends to become more conservative, potentially leading to missed detections.

Future work could involve exploring unsupervised domain adaptation or self-supervised pre-training to further bridge the gap between different clinical environments without requiring massive amounts of newly labeled data for every new hospital.

Summary updated

Abstract

Precise localization of pathologies in endoscopy is a critical task in medical imaging, with deep learning (DL) frameworks now serving as the primary methodology for automated lesion detection. Today, generic object detection methods, such as YOLO, provide state-of-the-art performance in this context. However, their robustness often decreases under real-world clinical conditions due to variations across endoscopic devices, intestinal content, dataset bias, and the presence of small or low-contrast lesions. Motivated by these challenges, we propose a novel polyp detection framework based on a Attention Residual U-Net offering enhanced polyp localization. Instead of directly regressing bounding boxes from global feature representations, the proposed method learns lesion-focused spatial representations and converts them into bounding-box predictions during inference. Unlike conventional encoder-decoder localization frameworks that directly propagate all encoder activations into the decoding pathway, the proposed method selectively suppresses irrelevant background responses before feature reconstruction. This encourages the network to focus on lesion-relevant spatial evidence under heterogeneous imaging conditions. To achieve this, the framework integrates residual feature propagation, attention-guided encoder filtering, and multi-scale contextual learning. The proposed framework was evaluated on multiple public endoscopic datasets acquired using different devices. The comparative experimental results validate its effectiveness over the relevant state-of-the-art methods, with an improvement up to 22.0% in terms of the area under the receiving operating characteristic (AUC).
BibTeX
@misc{imtiaz2026attention,
  author = {Romil Imtiaz and Dimitris K. Iakovidis},
  title = {{Attention Residual U-Net for Enhanced Cross-Dataset Polyp Localization in Endoscopy}},
  year = {2026},
  eprint = {2611.02726},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02726},
}
Figure