CFCG: Semi-Supervised Semantic Segmentation via Cross-Fusion and Contour Guidance Supervision

Shuo Li, Yue He, Weiming Zhang, Wei Zhang 0197, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang

Abstract

Semi-supervised semantic segmentation (SSSS) leverages a small amount of labeled data alongside a large pool of unlabeled data to reduce the high cost of pixel-level annotation. However, existing methods often suffer from "confirmation bias" due to noisy pseudo-labels and a reliance on pixel-wise confidence thresholds that fail near object boundaries. We propose CFCG, a framework that employs Cross-Fusion of multiple learners and Contour Guidance to refine pseudo-labels. By treating semantic contours as proxies for uncertainty, CFCG effectively mitigates error accumulation. Our method achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes benchmarks.

Introduction

Semantic segmentation requires meticulous manual labeling, making it one of the most expensive computer vision tasks. SSSS aims to alleviate this by training models on limited labeled images and a larger set of unlabeled data, typically using consistency regularization. In this setup, a model's own high-confidence predictions serve as "pseudo-labels" for unlabeled images.

Despite its success, SSSS faces two major hurdles:

  1. Noisy Supervision: A single model's predictions are often biased, leading to error reinforcement.
  2. The Confidence Trap: Standard methods trust pseudo-labels based on pixel-wise confidence scores. However, models are often "confidently wrong," particularly at object boundaries.
Confidence scores vs. error distribution. High-confidence errors frequently occur at object boundaries.

As shown in the visualization above, a significant portion of high-confidence predictions can be incorrect, especially near contours. This demonstrates that simple thresholding is insufficient for robust semi-supervised learning.

Methodology

The CFCG framework addresses these challenges through three core components:

Coherent Perturbation (CP)

To enhance robustness, CFCG applies "heavy" perturbations at both the image level (e.g., noise, blur, erasure) and the feature level within the neural network's layers. This forces the model to learn more invariant and discriminative features.

Cross-Fusion Supervision (CFS)

Instead of a single teacher-student setup, CFCG utilizes multiple learners. It employs a channel-wise attention mechanism to fuse predictions from these learners. This "wisdom of the crowd" approach generates higher-quality pseudo-labels by tapping into the diverse knowledge captured by different model branches.

Adaptive Contour Guidance Module (ACGM)

The ACGM is the framework's most novel contribution. Recognizing that errors cluster at object boundaries, ACGM extracts semantic contours from pseudo-labels using a Laplacian operator. It then generates a weight map that directs the model's attention toward these error-prone regions. This guidance is introduced progressively: as training advances and pseudo-labels become more reliable, the model relies more heavily on these refined contour cues.

The CFCG architecture, including the Adaptive Contour Guidance Module (ACGM) and Cross-Fusion Module (CFM).

Experiments and Results

We evaluated CFCG on two industry-standard benchmarks: PASCAL VOC 2012 (20 categories) and Cityscapes (19 categories for urban scenes). Experiments were conducted under various partition protocols (e.g., 1/8 or 1/16 labels) to simulate data scarcity.

Key Results

CFCG established new state-of-the-art (SOTA) benchmarks:

  • PASCAL VOC 2012: Under the 1/8 labeling protocol, it achieved 77.10% mIoU with a single learner and 77.55% with fusion inference, outperforming previous leaders like CPS and UCC by over 2%.
  • Cityscapes: Under the 1/8 protocol, it reached 78.49% mIoU, a gain of nearly 1% over previous methods.

Crucially, even the single-learner version of CFCG (without the extra computation of fusion inference at test time) outperforms previous SOTA methods, making it highly practical for real-world deployment.

Discussion and Conclusion

The success of CFCG highlights that confidence is not always a reliable indicator of accuracy. By using semantic contours as proxies for uncertainty, the model can better navigate the "confidence trap." Furthermore, the cross-fusion of multiple learners provides a more robust supervision signal than single-model approaches.

While fusion inference provides the best results, it does increase computational costs during deployment. Future work could explore extending contour guidance to other dense prediction tasks, such as depth estimation or instance segmentation, where boundary accuracy is equally critical.

Summary updated

Abstract

Current state-of-the-art semi-supervised semantic segmentation (SSSS) methods typically adopt pseudo labeling and consistency regularization between multiple learners with different perturbations. Although the performance is desirable, many issues remain: (1) supervisions from a single learner tend to be noisy which causes unreliable consistency regularization (2) existing pixel-wise confidence-score-based reliability measurement causes potential error accumulation as the training proceeds. In this paper, we propose a novel SSSS framework, called CFCG, which combines cross-fusion and contour guidance supervision to tackle these issues. Concretely, we adopt both image-level and feature-level perturbations to expand feature distribution thus pushing the potential limits of consistency regularization. Then, two particular modules are proposed to enable effective semi-supervised learning under heavy coherent perturbations. Firstly, Cross-Fusion Supervision (CFS) mechanism leverages multiple learners to enhance the quality of pseudo labels. Secondly, we introduce an adaptive contour guidance module (ACGM) to effectively identify unreliable spatial regions in pseudo labels. Finally, our proposed CFCG achieves gains of mIoU +1.40%, +0.89% with a single learner and +1.85%, +1.33% by fusion inference on PASCAL VOC 2012 and on Cityscapes respectively under 1/8 protocols, clearly surpassing previous methods and reaching the state-of-the-art.
BibTeX
@inproceedings{li2023cfcg,
  author = {Shuo Li and Yue He and Weiming Zhang and Wei Zhang and Xiao Tan and Junyu Han and Errui Ding and Jingdong Wang},
  title = {{CFCG: Semi-Supervised Semantic Segmentation via Cross-Fusion and Contour Guidance Supervision}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01498},
}
Figure