CARD: Semi-supervised Semantic Segmentation via Class-agnostic Relation based Denoising

Xiaoyang Wang, Jimin Xiao, Bingfeng Zhang, Limin Yu

Abstract

Semi-supervised semantic segmentation often relies on pseudo-labeling, which is prone to confirmation bias due to noisy initial predictions. We present CARD (Class-agnostic Relation based Denoising), a framework that mitigates this issue by leveraging semantic connections between pixels. By employing a Class-agnostic Relation Network (CARNet), CARD identifies pixels belonging to the same semantic group regardless of their category. This allows the model to "borrow" high-confidence predictions from reliable regions to correct uncertain ones. CARD achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes, significantly outperforming standard self-training baselines.

Introduction

Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to produce. Semi-supervised learning (SSL) addresses this by utilizing a small labeled dataset alongside a larger unlabeled one. A common SSL technique is pseudo-labeling, where a model generates labels for unlabeled data to use as training targets. However, early-stage models often produce noisy labels. Training on these errors leads to confirmation bias, where the model reinforces its own mistakes. While heavy data augmentation can mitigate this, directly denoising these pseudo-labels remains a challenging but effective alternative.

Visualizing the relation-based denoising process: (a) input image, (b) initial noisy pseudo-label, (c) initial relation map, (d) enhanced relation map, and (e) corrected pseudo-label.

Methods

The CARD framework introduces a three-part strategy to refine pseudo-labels progressively, ensuring that the model learns from increasingly accurate supervision.

1. Class-agnostic Relation Network (CARNet)

Standard feature encoders often struggle to relate pixels that are spatially distant or visually distinct but belong to the same object. CARNet is trained to map features into a space where the distance represents semantic similarity. Crucially, it is class-agnostic—it focuses on whether pixels belong to the same entity rather than their specific category. This decoupling prevents the network from inheriting the classification biases of the main model.

2. Class-balanced Reliability Estimation

To account for varying learning difficulties across different classes, CARD maintains a "confidence bank." Instead of a global threshold, it uses dynamic, class-specific thresholds to identify "reliable" (high confidence) and "noisy" (low confidence) regions in a class-balanced manner.

3. Progressive Label Denoising

For every noisy pixel, the system uses CARNet to find its semantic neighbors among the reliable pixels. It then performs a weighted ensemble—essentially a "vote"—where the reliable neighbors inform the noisy pixel's correct label. This process follows a curriculum, starting with the most obvious errors and gradually refining more confident predictions as training progresses.

The overall CARD framework, illustrating the interaction between the Teacher/Student models, the CARNet, and the progressive denoising pipeline.

Experimental Results

CARD was evaluated on the PASCAL VOC 2012 and Cityscapes benchmarks using a ResNet-101 backbone. The results demonstrate significant improvements over existing methods:

  • Performance Gap: On PASCAL VOC with 1/8 of the labels, CARD reached 74.07% mIoU, nearly matching the 74.50% achieved by a fully supervised model.
  • State-of-the-Art Comparison: On the 1/50 split of PASCAL VOC, CARD outperformed the previous state-of-the-art (Contra-SemiSeg) by 2.7%.
  • Ablation Insights: Adding the CARNet-based denoising component alone provided a 7.3% boost in accuracy compared to standard self-training baselines.

Conclusion and Future Work

CARD demonstrates that leveraging class-agnostic semantic relations is a powerful tool for denoising pseudo-labels in semi-supervised semantic segmentation. By effectively addressing the confirmation bias inherent in self-training, CARD sets new performance standards. Future research will focus on reducing the computational complexity of generating relation matrices and exploring the synergy between CARD and advanced data augmentation techniques like CutMix.

Summary updated

Abstract

Recent semi-supervised semantic segmentation methods focus on mining extra supervision from unlabeled data by generating pseudo labels. However, noisy labels are inevitable in this process which prevent effective self-supervision. This paper proposes that noisy labels can be corrected based on semantic connections among features. Since a segmentation classifier produces both high and low-quality predictions, we can trace back to feature encoder to investigate how a feature in a noisy group is related to those in the confident groups. Discarding the weak predictions from the classifier, rectified predictions are assigned to the wrongly predicted features through the feature relations. The key to such an idea lies in mining reliable feature connections. With this goal, we propose a class-agnostic relation network to precisely capture semantic connections among features while ignoring their semantic categories. The feature relations enable us to perform effective noisy label corrections to boost self-training performance. Extensive experiments on PASCAL VOC and Cityscapes demonstrate the state-of-the-art performances of the proposed methods under various semi-supervised settings.
BibTeX
@inproceedings{wang2022card,
  author = {Xiaoyang Wang and Jimin Xiao and Bingfeng Zhang and Limin Yu},
  title = {{CARD: Semi-supervised Semantic Segmentation via Class-agnostic Relation based Denoising}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/202},
}
Figure