Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning

Yingling Lu, Yijun Yang, Zhaohu Xing, Qiong Wang, Lei Zhu

Introduction

Colorectal cancer remains a leading cause of mortality worldwide, making early detection through colonoscopy a critical clinical priority. However, the automated segmentation of polyps in video feeds is a formidable challenge. Polyps often exhibit "high camouflage," possessing colors, textures, and boundaries nearly identical to surrounding healthy tissue. Furthermore, the temporal complexity of colonoscopy videos—marked by rapid camera movements and redundant information—makes maintaining consistent segmentation across frames difficult for traditional models.

To address these issues, the authors introduce Diff-VPS, the first framework to adapt Diffusion Probabilistic Models (DPMs) specifically for video polyp segmentation. By combining a multi-task learning strategy with a self-supervised temporal reasoning module, Diff-VPS overcomes the limitations of static image segmentation and sets a new state-of-the-art (SOTA) on benchmark datasets.

Methodology

Diff-VPS is built upon two primary architectural innovations designed to handle the unique demands of medical video analysis.

1. Multi-task Diffusion Model (MDM)

Standard diffusion models applied to segmentation often focus heavily on pixel-level details, which can lead to errors when the target object is camouflaged. To provide the model with better context, the authors introduced auxiliary tasks into the diffusion process:

  • Classification: Predicting the specific type of polyp (e.g., hyperplastic vs. invasive carcinoma).
  • Detection: Predicting a bounding box around the lesion.

These tasks act as high-level semantic "anchors." By learning to detect and classify while segmenting, the model gains a more holistic understanding of the scene, allowing it to better distinguish the polyp from the background during the iterative denoising process.

2. Temporal Reasoning Module (TRM)

To capture the dynamics of video data, the TRM utilizes a self-supervised proxy task: it attempts to reconstruct the current frame based on information from previous frames. This process is governed by an adversarial learning framework where the TRM acts as a generator and a pre-trained InceptionV3 network serves as a discriminator. This ensures that the model captures meaningful motion and texture cues. The resulting spatiotemporal priors are then fused with spatial features to condition the diffusion process, ensuring temporal consistency across the video sequence.

Experimental Setup and Results

The model was evaluated using SUN-SEG, the largest and most comprehensive video polyp dataset available, containing 113 videos. The dataset is categorized into "Easy" and "Hard" subsets, further divided into "Seen" (familiar scenarios) and "Unseen" (novel cases) to test the model's generalization capabilities.

Performance Metrics

Diff-VPS demonstrated superior performance over existing SOTA methods like PNS+ and AutoSAM:

  • SUN-SEG-Hard (Seen): Achieved a Dice score of 0.868, outperforming PNS+ by 1.3%.
  • SUN-SEG-Easy (Unseen): Achieved a Dice score of 0.767, significantly surpassing AutoSAM by 1.4%.

Beyond simple overlap metrics, the model showed notable improvements in structural similarity measures (S-measure and Enhanced-alignment measure). This indicates that Diff-VPS produces more anatomically accurate shapes rather than fragmented pixel clusters, a crucial requirement for clinical utility.

Key Insights

  • Diffusion as a Discriminator: The research demonstrates that the iterative denoising process of diffusion models is exceptionally well-suited for grasping the intrinsic structure and texture of camouflaged objects.
  • Multi-tasking Benefits: By incorporating object-level tasks (detection and classification), the model compensates for the scarcity of pixel-level annotations, leading to more balanced and robust feature extraction.
  • Reconstruction for Reasoning: Forcing the model to reconstruct frames provides a deeper "understanding" of temporal dynamics compared to simpler methods like optical flow.

Conclusion and Future Work

Diff-VPS represents a significant advancement in the application of diffusion models to medical imaging. While the model achieves SOTA accuracy, the iterative nature of diffusion models presents a challenge for real-time deployment during live colonoscopies. Future research will likely focus on optimizing the sampling process to increase inference speed and testing the framework's versatility on other types of endoscopic and surgical video data.

Summary updated

BibTeX
@inproceedings{lu2024diff,
  author = {Yingling Lu and Yijun Yang and Zhaohu Xing and Qiong Wang and Lei Zhu},
  title = {{Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72089-5\_16},
}
Figure