Abstract
Semantic segmentation models traditionally rely on labor-intensive, pixel-perfect "fine" annotations. While "coarse" labels (rough polygons) are significantly cheaper to produce, they introduce substantial label noise, especially at object boundaries, leading to the "High-Loss Dilemma." This paper introduces the Adversarial Reweighting Module (ARM), a 53-parameter plugin that leverages prediction variance to differentiate between informative hard samples and detrimental mislabeled noise. Through an adversarial training framework, ARM dynamically reweights pixel-level losses, allowing models to learn effectively from noisy datasets. Evaluations on Cityscapes and ADE20K demonstrate that ARM consistently outperforms standard training and existing hard-example mining techniques.
Introduction: The High-Loss Dilemma
The primary bottleneck in semantic segmentation is the cost of data annotation. For instance, a single image in the Cityscapes dataset requires approximately 1.5 hours of manual labor for "fine" labeling. In contrast, "coarse" annotations take only 7 minutes but are riddled with errors, particularly at object boundaries where background pixels are frequently mislabeled as foreground.
Standard training strategies often struggle with these noisy labels due to the High-Loss Dilemma. In clean datasets, a high loss value typically signifies a "hard" sample—a complex feature that the model needs to learn. However, in coarse datasets, high loss often indicates a "wrong" sample—a labeling error. Traditional methods like Online Hard Example Mining (OHEM) focus on these high-loss pixels, inadvertently causing the model to overfit to noise. Conversely, robust loss functions that ignore high-loss regions may fail to capture the genuine hard pixels necessary for sharp boundary definitions.
Methodology
The authors propose a confidence-based reweighting strategy that uses the variance of model predictions as a proxy for sample value.
Variance as a Confidence Indicator
Instead of relying on the absolute loss value, the model examines the variance of class-likelihood predictions.
- High Variance (Low Confidence): The model is uncertain, with probabilities spread across multiple classes. These are often genuine "hard" samples.
- Low Variance (High Confidence): The model is certain about its prediction. If a high-confidence prediction contradicts the provided label (resulting in high loss), it is highly likely that the label itself is incorrect.
Adversarial Reweighting Module (ARM)
The core of the proposed method is the ARM, a lightweight multilayer perceptron (MLP) with only 53 parameters. The training process is structured as an adversarial game:
- The Segmentation Model aims to minimize the weighted loss to master the segmentation task.
- The ARM aims to maximize the weighted loss by assigning higher weights to low-confidence (high-variance) regions.
This adversarial setup forces the segmentation model to prioritize regions where it is most uncertain, effectively "mining" for valuable information while suppressing the influence of mislabeled high-confidence pixels. A critical technical detail is the "detachment" of the gradient during variance calculation, preventing the segmentation model from artificially inflating variance to minimize its own loss.
Experimental Results
The ARM was evaluated on the Cityscapes and ADE20K benchmarks, demonstrating significant improvements in robustness and accuracy.
- Cityscapes Coarse: ARM improved the mean Intersection over Union (mIoU) by 1.61% over the baseline. Notably, OHEM actually degraded performance on this dataset due to its tendency to overfit to label noise.
- ADE20K: Integrated with DeepLabv3+, ARM achieved a state-of-the-art mIoU of 47.50.
- Efficiency: Due to its minimal parameter count and operation on final likelihood maps, ARM adds negligible computational overhead and remains model-agnostic.
Conclusion
The Adversarial Reweighting Module provides a simple yet effective solution for training segmentation models on noisy, coarse-labeled data. By shifting the focus from high-loss pixels to low-confidence pixels, ARM successfully navigates the High-Loss Dilemma. While a performance gap still exists between models trained on coarse versus fine data, ARM significantly narrows this divide. Future work may involve combining ARM with active label correction techniques to further refine boundary accuracy.
Summary updated