SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation

Qiang Hu, Zhenyu Yi, Ying Zhou, Fang Peng, Mei Liu, Qiang Li, Zhiwei Wang

Abstract

Colonoscopy videos provide richer information for polyp segmentation in rectal cancer diagnosis compared to static images. However, rapid endoscopic movement and close-up observations often lead to spatial incoherence and sequences of low-quality frames, limiting the accuracy of current segmentation methods. This paper introduces the SALI network, a robust video polyp segmentation framework that combines a Short-term Alignment Module (SAM) and a Long-term Interaction Module (LIM). SAM utilizes deformable convolutions to align features across adjacent frames, capturing stable short-term representations. LIM maintains a historical memory bank of polyp representations, using retrospective relations to reconstruct reliable features during periods of low image quality. Benchmarking on the SUN-SEG dataset demonstrates that SALI significantly outperforms existing state-of-the-art methods, improving Dice scores by up to 4.1% on unseen datasets.

Introduction

Colorectal cancer remains a significant global health challenge, but early detection and removal of polyps during colonoscopy can drastically improve patient outcomes. While automated polyp segmentation has advanced significantly with deep learning, transitioning from static image analysis to video processing introduces unique complexities.

In clinical colonoscopy, the environment is highly dynamic. Unlike standard video datasets where objects move against a stable background, colonoscopy involves a moving camera traversing a deformable tubular structure. This results in three primary difficulties:

  1. Unreliable Optical Flow: Traditional motion-tracking methods fail because the reflective, wet surfaces of the gut produce noisy pixel-level data.
  2. Rapid Spatial Variations: Sudden camera jerks or close-up views cause drastic visual shifts between consecutive frames.
  3. Persistent Low Quality: Factors such as motion blur, specular reflection (glare), and debris can obscure polyps for multiple frames, causing models to lose track of the target.

The SALI Framework

To address these challenges, the SALI network adopts a transformer-based backbone (PVTv2) and introduces two specialized modules designed to maintain temporal consistency and feature reliability.

Short-term Alignment Module (SAM)

The SAM is designed to harmonize features between the current frame and its immediate predecessor. Rather than relying on error-prone optical flow, SAM employs Deformable Convolutions (DConv). This approach allows the network to "warp" the features of the previous frame to align with the current spatial layout. By learning these offsets, the model creates a stabilized short-term representation that is resilient to the rapid spatial changes common in endoscopic procedures.

Long-term Interaction Module (LIM)

While SAM handles frame-to-frame transitions, the Long-term Interaction Module (LIM) provides a broader temporal context. LIM functions as a memory bank, storing up to 35 historical polyp representations sampled at regular intervals (e.g., every 5 frames).

When the current frame suffers from low visual quality—such as a sudden blur or glare—the LIM utilizes a Masked-Attention (MA) mechanism. This allows the model to look back at "reliable" historical views of the polyp to help reconstruct the current frame's features. This "retrospective interaction" ensures that the segmentation remains stable even when the immediate visual cues are weak. The memory bank is managed dynamically, removing the least relevant historical data as new frames are added to maintain computational efficiency.

Experimental Results

The SALI network was evaluated on the SUN-SEG dataset, the largest and most diverse benchmark for video polyp segmentation. The dataset includes categories for "Seen" and "Unseen" clips, further divided into "Easy" and "Hard" scenarios to test both accuracy and generalization.

Quantitative Performance

SALI demonstrated superior performance across all test subsets compared to previous state-of-the-art models like PNS+ and SLT-Net. The improvements in the Dice score—a critical metric for segmentation overlap—were notable:

  • Seen-Easy: +2.1%
  • Seen-Hard: +2.5%
  • Unseen-Easy: +4.1%
  • Unseen-Hard: +1.9%

The significant boost in the "Unseen" categories is particularly important, as it indicates that the model generalizes well to new patients and different endoscopic equipment, a prerequisite for clinical deployment.

Qualitative Robustness

Qualitative analysis shows that SALI effectively eliminates the "flicker" effect seen in other models. While competing methods often lose the polyp mask during fast camera movements or lighting changes, SALI’s integration of short-term alignment and long-term memory allows it to maintain a consistent and accurate boundary.

Conclusion and Future Work

The SALI network represents a significant step forward in robust video polyp segmentation. By replacing traditional optical flow with deformable alignment and introducing a long-term interaction memory bank, the model overcomes the inherent noise and instability of colonoscopy footage.

Future research will focus on improving label efficiency. Currently, training requires extensive frame-by-frame manual annotation. Investigating semi-supervised or self-supervised learning techniques could allow the model to learn from the vast quantities of unlabeled colonoscopy videos available in clinical archives. Additionally, optimizing the architecture for real-time, low-latency performance will be essential for its integration into live surgical assistance systems.

Summary updated

Abstract

Colonoscopy videos provide richer information in polyp segmentation for rectal cancer diagnosis. However, the endoscope's fast moving and close-up observing make the current methods suffer from large spatial incoherence and continuous low-quality frames, and thus yield limited segmentation accuracy. In this context, we focus on robust video polyp segmentation by enhancing the adjacent feature consistency and rebuilding the reliable polyp representation. To achieve this goal, we in this paper propose SALI network, a hybrid of Short-term Alignment Module (SAM) and Long-term Interaction Module (LIM). The SAM learns spatial-aligned features of adjacent frames via deformable convolution and further harmonizes them to capture more stable short-term polyp representation. In case of low-quality frames, the LIM stores the historical polyp representations as a long-term memory bank, and explores the retrospective relations to interactively rebuild more reliable polyp features for the current segmentation. Combing SAM and LIM, the SALI network of video segmentation shows a great robustness to the spatial variations and low-visual cues. Benchmark on the large-scale SUNSEG verifies the superiority of SALI over the current state-of-the-arts by improving Dice by 2.1%, 2.5%, 4.1% and 1.9%, for the four test sub-sets, respectively. Codes are at https://github.com/Scatteredrain/SALI.
BibTeX
@inproceedings{hu2024sali,
  author = {Qiang Hu and Zhenyu Yi and Ying Zhou and Fang Peng and Mei Liu and Qiang Li and Zhiwei Wang},
  title = {{SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=l6k2YQPd9K},
}
Figure