Variance Loss: A Confidence-Based Reweighting Strategy for Coarse Semantic Segmentation

Jingchao Liu, Ye Du, Qingjie Liu, Yunhong WangarXiv 2020 · arXiv

Abstract

Semantic segmentation models traditionally rely on labor-intensive, pixel-perfect "fine" annotations. While "coarse" labels (rough polygons) are significantly cheaper to produce, they introduce substantial label noise, especially at object boundaries, leading to the "High-Loss Dilemma." This paper introduces the Adversarial Reweighting Module (ARM), a 53-parameter plugin that leverages prediction variance to differentiate between informative hard samples and detrimental mislabeled noise. Through an adversarial training framework, ARM dynamically reweights pixel-level losses, allowing models to learn effectively from noisy datasets. Evaluations on Cityscapes and ADE20K demonstrate that ARM consistently outperforms standard training and existing hard-example mining techniques.

Introduction: The High-Loss Dilemma

The primary bottleneck in semantic segmentation is the cost of data annotation. For instance, a single image in the Cityscapes dataset requires approximately 1.5 hours of manual labor for "fine" labeling. In contrast, "coarse" annotations take only 7 minutes but are riddled with errors, particularly at object boundaries where background pixels are frequently mislabeled as foreground.

Standard training strategies often struggle with these noisy labels due to the High-Loss Dilemma. In clean datasets, a high loss value typically signifies a "hard" sample—a complex feature that the model needs to learn. However, in coarse datasets, high loss often indicates a "wrong" sample—a labeling error. Traditional methods like Online Hard Example Mining (OHEM) focus on these high-loss pixels, inadvertently causing the model to overfit to noise. Conversely, robust loss functions that ignore high-loss regions may fail to capture the genuine hard pixels necessary for sharp boundary definitions.

Comparison of reweighting strategies based on loss versus confidence.

Methodology

The authors propose a confidence-based reweighting strategy that uses the variance of model predictions as a proxy for sample value.

Variance as a Confidence Indicator

Instead of relying on the absolute loss value, the model examines the variance of class-likelihood predictions.

  • High Variance (Low Confidence): The model is uncertain, with probabilities spread across multiple classes. These are often genuine "hard" samples.
  • Low Variance (High Confidence): The model is certain about its prediction. If a high-confidence prediction contradicts the provided label (resulting in high loss), it is highly likely that the label itself is incorrect.

Adversarial Reweighting Module (ARM)

The core of the proposed method is the ARM, a lightweight multilayer perceptron (MLP) with only 53 parameters. The training process is structured as an adversarial game:

  1. The Segmentation Model aims to minimize the weighted loss to master the segmentation task.
  2. The ARM aims to maximize the weighted loss by assigning higher weights to low-confidence (high-variance) regions.
The architecture of the Adversarial Reweighting Module (ARM) and its integration into the training pipeline.

This adversarial setup forces the segmentation model to prioritize regions where it is most uncertain, effectively "mining" for valuable information while suppressing the influence of mislabeled high-confidence pixels. A critical technical detail is the "detachment" of the gradient during variance calculation, preventing the segmentation model from artificially inflating variance to minimize its own loss.

Experimental Results

The ARM was evaluated on the Cityscapes and ADE20K benchmarks, demonstrating significant improvements in robustness and accuracy.

  • Cityscapes Coarse: ARM improved the mean Intersection over Union (mIoU) by 1.61% over the baseline. Notably, OHEM actually degraded performance on this dataset due to its tendency to overfit to label noise.
  • ADE20K: Integrated with DeepLabv3+, ARM achieved a state-of-the-art mIoU of 47.50.
  • Efficiency: Due to its minimal parameter count and operation on final likelihood maps, ARM adds negligible computational overhead and remains model-agnostic.

Conclusion

The Adversarial Reweighting Module provides a simple yet effective solution for training segmentation models on noisy, coarse-labeled data. By shifting the focus from high-loss pixels to low-confidence pixels, ARM successfully navigates the High-Loss Dilemma. While a performance gap still exists between models trained on coarse versus fine data, ARM significantly narrows this divide. Future work may involve combining ARM with active label correction techniques to further refine boundary accuracy.

Summary updated

Abstract

Coarsely-labeled semantic segmentation annotations are easy to obtain, but therefore bear the risk of losing edge details and introducing background pixels. Impeded by the inherent noise, existing coarse annotations are only taken as a bonus for model pre-training. In this paper, we try to exploit their potentials with a confidence-based reweighting strategy. To expand, loss-based reweighting strategies usually take the high loss value to identify two completely different types of pixels, namely, valuable pixels in noise-free annotations and mislabeled pixels in noisy annotations. This makes it impossible to perform two tasks of mining valuable pixels and suppressing mislabeled pixels at the same time. However, with the help of the prediction confidence, we successfully solve this dilemma and simultaneously perform two subtasks with a single reweighting strategy. Furthermore, we generalize this strategy into an Adversarial Reweighting Module (ARM) and prove its convergence strictly. Experiments on standard datasets shows our ARM can bring consistent improvements for both coarse annotations and fine annotations. Specifically, built on top of DeepLabv3+, ARM improves the mIoU on the coarsely-labeled Cityscapes by a considerable margin and increases the mIoU on the ADE20K dataset to 47.50.
BibTeX
@misc{liu2020variance,
  author = {Jingchao Liu and Ye Du and Qingjie Liu and Yunhong Wang},
  title = {{Variance Loss: A Confidence-Based Reweighting Strategy for Coarse Semantic Segmentation}},
  year = {2020},
  eprint = {2009.05205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.05205},
}
Figure