Abstract
Semi-supervised semantic segmentation often relies on pseudo-labeling, which is prone to confirmation bias due to noisy initial predictions. We present CARD (Class-agnostic Relation based Denoising), a framework that mitigates this issue by leveraging semantic connections between pixels. By employing a Class-agnostic Relation Network (CARNet), CARD identifies pixels belonging to the same semantic group regardless of their category. This allows the model to "borrow" high-confidence predictions from reliable regions to correct uncertain ones. CARD achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes, significantly outperforming standard self-training baselines.
Introduction
Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to produce. Semi-supervised learning (SSL) addresses this by utilizing a small labeled dataset alongside a larger unlabeled one. A common SSL technique is pseudo-labeling, where a model generates labels for unlabeled data to use as training targets. However, early-stage models often produce noisy labels. Training on these errors leads to confirmation bias, where the model reinforces its own mistakes. While heavy data augmentation can mitigate this, directly denoising these pseudo-labels remains a challenging but effective alternative.
Methods
The CARD framework introduces a three-part strategy to refine pseudo-labels progressively, ensuring that the model learns from increasingly accurate supervision.
1. Class-agnostic Relation Network (CARNet)
Standard feature encoders often struggle to relate pixels that are spatially distant or visually distinct but belong to the same object. CARNet is trained to map features into a space where the distance represents semantic similarity. Crucially, it is class-agnostic—it focuses on whether pixels belong to the same entity rather than their specific category. This decoupling prevents the network from inheriting the classification biases of the main model.
2. Class-balanced Reliability Estimation
To account for varying learning difficulties across different classes, CARD maintains a "confidence bank." Instead of a global threshold, it uses dynamic, class-specific thresholds to identify "reliable" (high confidence) and "noisy" (low confidence) regions in a class-balanced manner.
3. Progressive Label Denoising
For every noisy pixel, the system uses CARNet to find its semantic neighbors among the reliable pixels. It then performs a weighted ensemble—essentially a "vote"—where the reliable neighbors inform the noisy pixel's correct label. This process follows a curriculum, starting with the most obvious errors and gradually refining more confident predictions as training progresses.
Experimental Results
CARD was evaluated on the PASCAL VOC 2012 and Cityscapes benchmarks using a ResNet-101 backbone. The results demonstrate significant improvements over existing methods:
- Performance Gap: On PASCAL VOC with 1/8 of the labels, CARD reached 74.07% mIoU, nearly matching the 74.50% achieved by a fully supervised model.
- State-of-the-Art Comparison: On the 1/50 split of PASCAL VOC, CARD outperformed the previous state-of-the-art (Contra-SemiSeg) by 2.7%.
- Ablation Insights: Adding the CARNet-based denoising component alone provided a 7.3% boost in accuracy compared to standard self-training baselines.
Conclusion and Future Work
CARD demonstrates that leveraging class-agnostic semantic relations is a powerful tool for denoising pseudo-labels in semi-supervised semantic segmentation. By effectively addressing the confirmation bias inherent in self-training, CARD sets new performance standards. Future research will focus on reducing the computational complexity of generating relation matrices and exploring the synergy between CARD and advanced data augmentation techniques like CutMix.
Summary updated