Adaptive Affinity Loss and Erroneous Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation

Xiangrong Zhang, Zelin Peng, Peng Zhu, Tianyang Zhang, Chen Li, Huiyu Zhou, Licheng Jiao

Introduction

Semantic segmentation typically requires expensive, pixel-perfect annotations. Weakly Supervised Semantic Segmentation (WSSS) attempts to bypass this by using cheap image-level labels to train pixel-level models. Historically, the most accurate WSSS models use complex multi-stage pipelines to generate and refine initial pixel masks. Single-stage methods are more efficient but often suffer from noisy Class Activation Map (CAM) pseudo-labels. As illustrated in Figure 1, these initial predictions are frequently incomplete or include background noise, leading models to over-fit to "fake" pixels.

Comparison of initial CAM-based predictions versus ground truth labels showing the inherent noise in weak supervision.

Methodology

This research proposes a single-stage framework using a WideResNet38 backbone that refines pseudo-labels on the fly through two novel loss functions. This approach allows the model to achieve high accuracy without the computational burden of multi-stage training.

Adaptive Affinity (AA) Loss

Standard affinity learning encourages neighboring pixels with similar features to share the same label. However, this can propagate errors if the initial label is wrong. The AA loss modifies this by scaling connection weights based on the network's own activation confidence scores. By focusing affinity learning on reliable, high-confidence internal regions and penalizing connections near ambiguous boundaries, the model prevents the spread of erroneous semantics.

Label Reassign (LR) Loss

To correct fundamental errors in the initial pseudo-labels, the authors introduce a metric learning approach. The network calculates per-class centroids in the feature embedding space. During training, pixels are dynamically reassigned to the class of their nearest centroid. The LR loss then pushes pixel embeddings closer to these newly assigned centroids, allowing the network to filter out noisy labels and self-correct during the training run.

The proposed single-stage architecture featuring the Classification and Segmentation branches integrated with AA and LR losses.

Experimental Results

The method was evaluated on the PASCAL VOC 2012 benchmark, utilizing the Semantic Boundaries Dataset (SBD) for augmentation.

  • Accuracy: The proposed method achieves 63.9% mIoU on the validation set and 64.8% mIoU on the test set, outperforming other single-stage methods like RRM (62.9% test mIoU).
  • Ablation Impact: The baseline model starts at 60.0% validation mIoU. Adding adaptive affinity improves this to 63.2%, and the final inclusion of Label Reassign loss brings it to 63.9%.
  • Efficiency: The refinement process requires less than a 4% increase in training time compared to a standard single-stage baseline, while offering massive compute savings over multi-stage architectures.

Conclusion

By embedding refinement steps directly into the loss functions, this work demonstrates that single-stage WSSS can rival the performance of multi-stage pipelines. The combination of confidence-weighted propagation and dynamic embedding reassignment provides a robust solution to the problem of noisy pseudo-labels in weakly supervised learning.

Summary updated

BibTeX
@inproceedings{zhang2021adaptive,
  author = {Xiangrong Zhang and Zelin Peng and Peng Zhu and Tianyang Zhang and Chen Li and Huiyu Zhou and Licheng Jiao},
  title = {{Adaptive Affinity Loss and Erroneous Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475675},
}
Figure