Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentation

Juntong Fan, Shuyi Fan, Debesh Jha, Changsheng Fang, Tieyong Zeng, Hengyong Yu, Dayang WangarXiv 2025 · DOI · arXiv

Introduction

Colorectal cancer is a leading cause of mortality worldwide, yet it is largely preventable through the early detection and removal of precancerous polyps during colonoscopy procedures. Despite the potential of computer-aided diagnosis, clinical environments present significant challenges for AI: low contrast between lesions and healthy tissue, specular highlights from endoscopic lighting, and motion-induced blur. Existing deep learning models often struggle to generalize, frequently over-relying on local texture cues while failing to capture the broader topological structure of the intestinal mucosa.

The FOCUS-Med Architecture

To overcome these limitations, the researchers introduced FOCUS-Med (Fusion of spatial and structural graph with attentiOnal features for Context-aware polyp segmentation). This novel architecture moves beyond standard convolutional approaches by integrating graph-based reasoning and advanced attention mechanisms.

The core innovation lies in three primary components:

  1. Dual-GCN with Dijkstra Attention: FOCUS-Med utilizes a Dual Graph Convolutional Network to process information. While one graph captures local spatial details, the Structural Graph employs Dijkstra’s algorithm to calculate shortest paths between distant image regions. This allows the model to understand long-range anatomical relationships and "topological priors"—the overall shape and connectivity of the tissue—that are typically lost in standard convolutions.
  2. Location-Fused Stand-alone Self-Attention (LFSA): Standard self-attention often ignores the relative distance between pixels. LFSA explicitly incorporates row and column offsets into the attention calculation, helping the model more accurately delineate the irregular boundaries of polyps.
  3. Weighted Fast Normalized Fusion (WFNF): Rather than simple concatenation, WFNF uses trainable weights to learn the optimal importance of features from different network layers during the fusion process, reducing noise and improving representational power.
The FOCUS-Med architecture, illustrating the multi-stage ConvNext encoder and the integration of the Dual-GCN enhancement block with WFNF and LFSA modules for precise segmentation.

Datasets and Evaluation

The model was rigorously evaluated on four gold-standard public benchmarks: Kvasir-SEG, CVC-ClinicDB, Endoscene, and CVC-ColonDB. These datasets encompass a wide variety of polyp sizes, shapes, and lighting conditions.

In a pioneering move toward human-aligned AI, the authors also utilized GPT-4o to provide qualitative clinical assessments. The Large Language Model (LLM) was prompted to act as an expert reviewer, rating segmentation masks on criteria such as "Boundary Accuracy" and "Clinical Usefulness" using a 5-point Likert scale. This approach provides a more nuanced understanding of the model's performance than traditional pixel-overlap metrics alone.

Key Results and Discussion

FOCUS-Med achieved state-of-the-art results across all five primary metrics: Dice, IoU, MAE, Boundary F, and S-measure. On the CVC-ClinicDB dataset, the model reached a 94.74% Dice score, significantly outperforming previous leading models like DCRNet (89.24%).

Quantitative performance comparison of FOCUS-Med against existing state-of-the-art models across four benchmark datasets. Bold values indicate the best performance.

The LLM-based evaluation further highlighted the model's robustness. While previous state-of-the-art models showed a "heavy tail" of low scores on difficult cases, FOCUS-Med's ratings were tightly clustered in the "Excellent" range (4-5). Furthermore, the model maintains an inference speed of 10.63 FPS, making it a viable candidate for real-time clinical assistance.

Conclusion

By focusing on the structural topology of the image rather than just pixel intensity, FOCUS-Med demonstrates superior robustness to the glare and blur common in real-world colonoscopy footage. While the model still faces challenges with extremely small or texture-mimicking polyps, its combination of graph neural networks and human-aligned evaluation marks a significant advancement in the field of medical image segmentation.

Summary updated

Abstract

Accurate endoscopic image segmentation on the polyps is critical for early colorectal cancer detection. However, this task remains challenging due to low contrast with surrounding mucosa, specular highlights, and indistinct boundaries. To address these challenges, we propose FOCUS-Med, which stands for Fusion of spatial and structural graph with attentional context-aware polyp segmentation in endoscopic medical imaging. FOCUS-Med integrates a Dual Graph Convolutional Network (Dual-GCN) module to capture contextual spatial and topological structural dependencies. This graph-based representation enables the model to better distinguish polyps from background tissues by leveraging topological cues and spatial connectivity, which are often obscured in raw image intensities. It enhances the model's ability to preserve boundaries and delineate complex shapes typical of polyps. In addition, a location-fused stand-alone self-attention is employed to strengthen global context integration. To bridge the semantic gap between encoder-decoder layers, we incorporate a trainable weighted fast normalized fusion strategy for efficient multi-scale aggregation. Notably, we are the first to introduce the use of a Large Language Model (LLM) to provide detailed qualitative evaluations of segmentation quality. Extensive experiments on public benchmarks demonstrate that FOCUS-Med achieves state-of-the-art performance across five key metrics, underscoring its effectiveness and clinical potential for AI-assisted colonoscopy.
BibTeX
@misc{fan2025large,
  author = {Juntong Fan and Shuyi Fan and Debesh Jha and Changsheng Fang and Tieyong Zeng and Hengyong Yu and Dayang Wang},
  title = {{Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentation}},
  year = {2025},
  eprint = {2508.07028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07028},
}
Figure