SALI: Short-Term Alignment and Long-Term Interaction Network for Colonoscopy Video Polyp Segmentation

Qiang Hu, Zhenyu Yi, Ying Zhou, Fang Peng, Mei Liu, Qiang Li, Zhiwei Wang

TL;DR

SALI (Short-term Alignment and Long-term Interaction) is a new neural network designed to accurately segment polyps in colonoscopy videos, even when the camera moves fast or the image quality drops. By combining a spatial alignment module for adjacent frames with a long-term memory bank of past polyp appearances, it achieves state-of-the-art results on the large-scale SUN-SEG benchmark.

What Problem Does This Paper Address?

Automatic polyp segmentation is a vital tool for preventing colorectal cancer, yet colonoscopy videos present unique challenges that trip up standard computer vision models:

  1. Unreliable Optical Flow: In most videos, the camera is still while objects move. In colonoscopy, the camera moves through the digestive tract while the "objects" (polyps and tissue) are relatively stationary. This makes traditional motion-tracking (optical flow) ineffective.
  2. Rapid Spatial Variations: Close-up views and sudden camera jerks cause massive visual changes between even two consecutive frames.
  3. Persistent Low Quality: Glare, blur, and poor lighting can obscure a polyp for several seconds, causing models with "short memories" to lose track of the target.

Methods

The authors propose the SALI network, which uses a transformer-based backbone (PVTv2) and two novel modules to handle temporal data:

  • Short-term Alignment Module (SAM): Instead of relying on optical flow, SAM uses Deformable Convolutions (DConv) to align the features of the current frame with the previous one. It essentially "warps" the previous frame's features to match the current spatial layout, creating a stable short-term representation.
  • Long-term Interaction Module (LIM): This acts as the system's "memory." It maintains a bank of up to 35 historical polyp representations (sampled every 5 frames). When the current frame is low-quality, LIM uses a Masked-Attention (MA) block to look back at previous "reliable" views of the polyp to help reconstruct and segment the current one.

The model is "online," meaning it only looks at past and current frames, making it more suitable for potential real-time clinical use than "offline" models that require seeing the entire video at once.

Datasets

The researchers evaluated SALI on SUN-SEG, currently the largest and most comprehensive polyp video segmentation dataset. It contains 112 training clips and is divided into four testing categories:

  • Seen-Easy/Hard: Clips from the same videos used in training (but different segments).
  • Unseen-Easy/Hard: Clips from entirely different videos, testing the model's ability to generalize to new patients and environments.

Key Insights and Ideas

  • Alignment over Flow: The paper argues that for medical endoscopy, explicitly learning to align feature maps is superior to calculating pixel-level motion (optical flow), which is often too noisy in the gut's reflective environment.
  • The "Memory Bank" as a Buffer: The LIM module effectively "denoises" the current frame. If the current frame is blurry, the model "remembers" what the polyp looked like a few seconds ago and uses that information to maintain an accurate mask.
  • Dynamic Memory Management: To keep the memory bank efficient, the model uses a dynamic selection strategy, deleting the "least relevant" historical frames when the bank is full.

Key Results

SALI outperformed several state-of-the-art (SOTA) models, including PNS+ and SLT-Net, across all metrics.

  • Accuracy Boost: On the "Unseen-Easy" dataset, SALI improved the Dice score (a measure of overlap between the predicted mask and the ground truth) by 4.1% over the previous best method.
  • Robustness: Qualitative results show that while other models "flicker" or lose the polyp entirely during fast camera moves or glare, SALI maintains a stable, accurate segmentation mask.
  • Generalization: The high performance on "Unseen" datasets suggests the model isn't just memorizing the training data but has learned the fundamental visual characteristics of polyps.

Limitations and Future Work

While SALI shows impressive performance, the authors note a few areas for growth:

  • Label Efficiency: Training these models currently requires massive amounts of frame-by-frame manual annotations. The authors suggest investigating "label-friendly" versions that require fewer annotated frames.
  • Unlabeled Data: Future iterations could leverage the vast amounts of unlabeled colonoscopy video available in hospitals to improve the model's robustness through self-supervised learning.
  • Real-time Constraints: While the architecture is "online," the paper does not explicitly detail the frames-per-second (FPS) performance required for live surgical assistance.

Summary updated

Abstract

Colonoscopy videos provide richer information in polyp segmentation for rectal cancer diagnosis. However, the endoscope’s fast moving and close-up observing make the current methods suffer from large spatial incoherence and continuous low-quality frames, and thus yield limited segmentation accuracy. In this context, we focus on robust video polyp segmentation by enhancing the adjacent feature consistency and rebuilding the reliable polyp representation. To achieve this goal, we in this paper propose SALI network, a hybrid of Short-term Alignment Module (SAM) and Long-term Interaction Module (LIM). The SAM learns spatial-aligned features of adjacent frames via deformable convolution and further harmonizes them to capture more stable short-term polyp representation. In case of low-quality frames, the LIM stores the historical polyp representations as a long-term memory bank, and explores the retrospective relations to interactively rebuild more reliable polyp features for the current segmentation. Combing SAM and LIM, the SALI network of video segmentation shows a great robustness to the spatial variations and low-visual cues. Benchmark on the large-scale SUN-SEG verifies the superiority of SALI over the current state-of-the-arts by improving Dice by 2.1%, 2.5%, 4.1% and 1.9%, for the four test sub-sets, respectively. Codes are at https://github.com/Scatteredrain/SALI.
BibTeX
@inproceedings{hu2024sali,
  author = {Qiang Hu and Zhenyu Yi and Ying Zhou and Fang Peng and Mei Liu and Qiang Li and Zhiwei Wang},
  title = {{SALI: Short-Term Alignment and Long-Term Interaction Network for Colonoscopy Video Polyp Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=l6k2YQPd9K},
}

Loading related content...

Figure