Abstract
Colonoscopy videos provide richer information for polyp segmentation in rectal cancer diagnosis compared to static images. However, rapid endoscopic movement and close-up observations often lead to spatial incoherence and sequences of low-quality frames, limiting the accuracy of current segmentation methods. This paper introduces the SALI network, a robust video polyp segmentation framework that combines a Short-term Alignment Module (SAM) and a Long-term Interaction Module (LIM). SAM utilizes deformable convolutions to align features across adjacent frames, capturing stable short-term representations. LIM maintains a historical memory bank of polyp representations, using retrospective relations to reconstruct reliable features during periods of low image quality. Benchmarking on the SUN-SEG dataset demonstrates that SALI significantly outperforms existing state-of-the-art methods, improving Dice scores by up to 4.1% on unseen datasets.
Introduction
Colorectal cancer remains a significant global health challenge, but early detection and removal of polyps during colonoscopy can drastically improve patient outcomes. While automated polyp segmentation has advanced significantly with deep learning, transitioning from static image analysis to video processing introduces unique complexities.
In clinical colonoscopy, the environment is highly dynamic. Unlike standard video datasets where objects move against a stable background, colonoscopy involves a moving camera traversing a deformable tubular structure. This results in three primary difficulties:
- Unreliable Optical Flow: Traditional motion-tracking methods fail because the reflective, wet surfaces of the gut produce noisy pixel-level data.
- Rapid Spatial Variations: Sudden camera jerks or close-up views cause drastic visual shifts between consecutive frames.
- Persistent Low Quality: Factors such as motion blur, specular reflection (glare), and debris can obscure polyps for multiple frames, causing models to lose track of the target.
The SALI Framework
To address these challenges, the SALI network adopts a transformer-based backbone (PVTv2) and introduces two specialized modules designed to maintain temporal consistency and feature reliability.
Short-term Alignment Module (SAM)
The SAM is designed to harmonize features between the current frame and its immediate predecessor. Rather than relying on error-prone optical flow, SAM employs Deformable Convolutions (DConv). This approach allows the network to "warp" the features of the previous frame to align with the current spatial layout. By learning these offsets, the model creates a stabilized short-term representation that is resilient to the rapid spatial changes common in endoscopic procedures.
Long-term Interaction Module (LIM)
While SAM handles frame-to-frame transitions, the Long-term Interaction Module (LIM) provides a broader temporal context. LIM functions as a memory bank, storing up to 35 historical polyp representations sampled at regular intervals (e.g., every 5 frames).
When the current frame suffers from low visual quality—such as a sudden blur or glare—the LIM utilizes a Masked-Attention (MA) mechanism. This allows the model to look back at "reliable" historical views of the polyp to help reconstruct the current frame's features. This "retrospective interaction" ensures that the segmentation remains stable even when the immediate visual cues are weak. The memory bank is managed dynamically, removing the least relevant historical data as new frames are added to maintain computational efficiency.
Experimental Results
The SALI network was evaluated on the SUN-SEG dataset, the largest and most diverse benchmark for video polyp segmentation. The dataset includes categories for "Seen" and "Unseen" clips, further divided into "Easy" and "Hard" scenarios to test both accuracy and generalization.
Quantitative Performance
SALI demonstrated superior performance across all test subsets compared to previous state-of-the-art models like PNS+ and SLT-Net. The improvements in the Dice score—a critical metric for segmentation overlap—were notable:
- Seen-Easy: +2.1%
- Seen-Hard: +2.5%
- Unseen-Easy: +4.1%
- Unseen-Hard: +1.9%
The significant boost in the "Unseen" categories is particularly important, as it indicates that the model generalizes well to new patients and different endoscopic equipment, a prerequisite for clinical deployment.
Qualitative Robustness
Qualitative analysis shows that SALI effectively eliminates the "flicker" effect seen in other models. While competing methods often lose the polyp mask during fast camera movements or lighting changes, SALI’s integration of short-term alignment and long-term memory allows it to maintain a consistent and accurate boundary.
Conclusion and Future Work
The SALI network represents a significant step forward in robust video polyp segmentation. By replacing traditional optical flow with deformable alignment and introducing a long-term interaction memory bank, the model overcomes the inherent noise and instability of colonoscopy footage.
Future research will focus on improving label efficiency. Currently, training requires extensive frame-by-frame manual annotation. Investigating semi-supervised or self-supervised learning techniques could allow the model to learn from the vast quantities of unlabeled colonoscopy videos available in clinical archives. Additionally, optimizing the architecture for real-time, low-latency performance will be essential for its integration into live surgical assistance systems.
Summary updated