Abstract
Standard semantic segmentation models often fail when encountering real-world image corruptions such as noise or adverse weather. This research investigates how the internal mathematical structures of these models—specifically the choice of loss function—impact their robustness. We demonstrate that the common Softmax loss creates a rigid, low-dimensional representation that is highly sensitive to input perturbations. By introducing SCrIBE (Sigmoid Cross Entropy with Implicit Background Estimation), we foster a higher-dimensional representation that provides "wiggle room" for corrupted data, significantly improving robustness without sacrificing performance on clean images.
Introduction
Semantic segmentation is critical for safety-oriented applications like autonomous driving. However, models trained with standard cross-entropy loss and Softmax activation are notoriously brittle. When faced with "out-of-distribution" corruptions—like sensor noise or snow—their performance degrades sharply. This paper explores the hypothesis that the structural dependencies imposed by Softmax are a primary cause of this vulnerability.
The Softmax "Neglect" Problem
In a standard Softmax setup, class probabilities are forced to sum to one. This creates a zero-sum competition where increasing one class's confidence necessitates decreasing all others. As the number of classes increases, the model focuses heavily on the "winning" class, neglecting the internal representations of others. This leads to a collapsed, low-dimensional representation space where even small input corruptions can push a pixel's representation across a decision boundary.
Methodology: SCrIBE
To address these limitations, we propose SCrIBE, which combines two key components:
- Sigmoid Cross Entropy: Replaces the Softmax competition with independent binary detectors for each class.
- Implicit Background Estimation (IBE): Instead of learning a separate background filter, the background is treated as the mathematical complement of all foreground classes. This ensures that foreground detectors share a rich, common representation of the background while remaining decorrelated from each other.
Dimensionality and the "Wiggle Room" Hypothesis
A key finding of our work is that SCrIBE retains a higher-dimensional representation compared to Softmax or IBE alone. We analyzed this using "Accumulated Explained Variance" across the model's internal components.
This higher dimensionality provides "wiggle room." Input corruption typically applies an affine transformation to the data in the model's internal space. In a high-dimensional space, these shifted representations are less likely to collide with the decision boundaries of other classes, making the model more robust to "messy" data.
Experimental Results
We evaluated our approach on the PASCAL VOC 2012 dataset, using the ImageNet-C toolkit to simulate 15 types of corruption across 5 severity levels.
Quantitative Performance
SCrIBE achieved an aggregate mIOU of 42.1 across all corruptions, a significant improvement over the Softmax Baseline (37.5) and IBE (40.3). Crucially, SCrIBE maintained a high mIOU on clean data (69.1 vs. 69.9 for the baseline), demonstrating that robustness does not require a trade-off in accuracy.
Qualitative Robustness
The visual impact of SCrIBE's robustness is most evident under severe corruption.
As shown in the figure above, while the baseline model's predictions fragment or disappear entirely under noise and blur, SCrIBE retains the general structure of the objects, demonstrating its superior ability to handle real-world sensor degradation.
Conclusion
Our research highlights that the way we structure class dependencies in neural networks profoundly affects their robustness. By moving away from the rigid competition of Softmax toward the more flexible, higher-dimensional representation of SCrIBE, we can build semantic segmentation models that are far better equipped for the unpredictable conditions of the real world.
Summary updated