TL;DR
This paper introduces a novel approach to video polyp segmentation that drastically reduces annotation costs by requiring only the first frame of a video to be labeled. Their "Propagative and Semantic Dual-Teacher Network" (PSDNet) leverages a unique dual-teacher system and a novel "back-propagation scoring" method to generate accurate pseudo-labels for unlabeled frames, achieving performance competitive with fully supervised methods.
What Problem Does This Paper Address?
Colorectal cancer (CRC) screening heavily relies on identifying polyps, and automated polyp segmentation is crucial for this. While image-based segmentation methods exist, they often fail to leverage the temporal information present in colonoscopy videos. Video polyp segmentation (VPS) methods aim to use this temporal data, but they are severely hampered by the extremely high cost of frame-by-frame pixel-wise annotations required for training. Existing semi-supervised methods offer some reduction in annotation burden, but still require labeling dozens of frames per video. This paper tackles the problem of excessive annotation cost in VPS by proposing a new task: First-Frame Supervised Video Polyp Segmentation (FSVPS), where only the very first frame of a video needs manual annotation, regardless of its length.
Methods
The core of the proposed solution is the Propagative and Semantic Dual-Teacher Network (PSDNet), built upon a teacher-student framework.
- Propagative Teacher: This teacher utilizes an off-the-shelf Universal Video Object Segmentation (UVOS) model (specifically XMem) to propagate the first-frame annotation to subsequent frames. This provides initial pseudo-labels but is prone to accumulating tracking errors over time.
- Semantic Teacher: To combat the errors of the propagative teacher, a semantic teacher is introduced. This teacher is an Exponential Moving Average (EMA) of the student model's weights. This EMA approach creates a more stable and time-invariant pseudo-label generator, capturing semantic understanding of polyps.
- Dual-Teacher Merging: The key innovation lies in how the outputs of these two teachers are merged. A novel back-propagation scoring method is employed. This method assesses the quality of a teacher's pseudo-label by propagating it backward to the first frame using the UVOS model. Pseudo-labels that align well with the first-frame ground truth after this backward propagation are considered high-quality. The method then adaptively selects the pseudo-label from the teacher with the higher quality score for each unlabeled frame to supervise the student model.
- Student Model: The student model is trained using a combination of the first-frame ground truth (supervised loss) and the merged pseudo-labels from the dual-teacher system (unsupervised loss).
Datasets
- SUN-SEG: This is the primary dataset used for training and evaluation. It's described as the largest polyp video segmentation dataset. The training set consists of 112 video clips with 19,544 frames. The test set is divided into four subsets: SUN-SEG-Seen-Easy, SUN-SEG-Seen-Hard, SUN-SEG-Unseen-Easy, and SUN-SEG-Unseen-Hard. Crucially, for training, only the first frame's annotation is provided per video.
Key Insights and Ideas
- First-Frame Supervision is Feasible: The paper demonstrates that it's possible to achieve competitive performance in video polyp segmentation with only a single frame annotation, significantly reducing annotation costs.
- Complementary Strengths of Teachers: The propagative teacher excels at initial tracking but suffers from error accumulation, while the semantic teacher (EMA of the student) provides stable, time-invariant predictions. Combining them is more effective than either alone.
- Back-Propagation Scoring for Quality Assessment: The idea of propagating pseudo-labels backward to the first frame to assess their quality is a clever way to leverage the initial annotation and temporal consistency. This allows for intelligent merging of teacher outputs.
- Video-Based Methods Still Reign: Even with significant annotation reduction, video-based methods (like PSDNet) generally outperform image-based methods in VPS due to their ability to exploit temporal information.
Key Results
- Competitive Performance: PSDNet achieves performance comparable to fully supervised methods on the SUN-SEG dataset, despite using only approximately 1/175 of the labeled data.
- State-of-the-Art Superiority: PSDNet outperforms other state-of-the-art semi-supervised methods, achieving an average improvement of 4.5% in Dice score across the four test subsets.
- Significant Gains from Components: Ablation studies show that adding the propagative teacher improves performance from a baseline of 65.2% to 81.4% Dice. The dual-teacher setup with back-propagation scoring further boosts performance to 89.3% (train set Dice) and 84.1% (test set Dice), demonstrating the effectiveness of each component.
- Qualitative Improvements: Visualizations show PSDNet producing more accurate segmentations and maintaining better temporal consistency, even in challenging scenarios with motion blur and light noise.
Limitations and Future Work
- UVOS Model Dependency: The performance of the propagative teacher is inherently linked to the quality of the underlying UVOS model (XMem). Errors in XMem can still propagate.
- Potential for Further Improvement: While PSDNet significantly reduces annotation cost, the authors suggest that its framework is adaptable to other medical video segmentation tasks where data scarcity is an issue, indicating potential for broader clinical applications. The paper also hints at the potential of Universal Video Object Segmentation (UVOS) models, but notes that training-based methods still hold an advantage for specific tasks like polyp segmentation.
Summary updated