On the Structures of Representation for the Robustness of Semantic Segmentation to Input Corruption

Charles Lehman, Dogancan Temel, Ghassan Alregib

Abstract

Standard semantic segmentation models often fail when encountering real-world image corruptions such as noise or adverse weather. This research investigates how the internal mathematical structures of these models—specifically the choice of loss function—impact their robustness. We demonstrate that the common Softmax loss creates a rigid, low-dimensional representation that is highly sensitive to input perturbations. By introducing SCrIBE (Sigmoid Cross Entropy with Implicit Background Estimation), we foster a higher-dimensional representation that provides "wiggle room" for corrupted data, significantly improving robustness without sacrificing performance on clean images.

Introduction

Semantic segmentation is critical for safety-oriented applications like autonomous driving. However, models trained with standard cross-entropy loss and Softmax activation are notoriously brittle. When faced with "out-of-distribution" corruptions—like sensor noise or snow—their performance degrades sharply. This paper explores the hypothesis that the structural dependencies imposed by Softmax are a primary cause of this vulnerability.

The Softmax "Neglect" Problem

In a standard Softmax setup, class probabilities are forced to sum to one. This creates a zero-sum competition where increasing one class's confidence necessitates decreasing all others. As the number of classes increases, the model focuses heavily on the "winning" class, neglecting the internal representations of others. This leads to a collapsed, low-dimensional representation space where even small input corruptions can push a pixel's representation across a decision boundary.

Methodology: SCrIBE

To address these limitations, we propose SCrIBE, which combines two key components:

  1. Sigmoid Cross Entropy: Replaces the Softmax competition with independent binary detectors for each class.
  2. Implicit Background Estimation (IBE): Instead of learning a separate background filter, the background is treated as the mathematical complement of all foreground classes. This ensures that foreground detectors share a rich, common representation of the background while remaining decorrelated from each other.

Dimensionality and the "Wiggle Room" Hypothesis

A key finding of our work is that SCrIBE retains a higher-dimensional representation compared to Softmax or IBE alone. We analyzed this using "Accumulated Explained Variance" across the model's internal components.

Figure 2: Accumulated Explained Variance across different models. SCrIBE (green) maintains a higher-dimensional representation compared to IBE (orange) and the Baseline (blue).

This higher dimensionality provides "wiggle room." Input corruption typically applies an affine transformation to the data in the model's internal space. In a high-dimensional space, these shifted representations are less likely to collide with the decision boundaries of other classes, making the model more robust to "messy" data.

Experimental Results

We evaluated our approach on the PASCAL VOC 2012 dataset, using the ImageNet-C toolkit to simulate 15 types of corruption across 5 severity levels.

Quantitative Performance

SCrIBE achieved an aggregate mIOU of 42.1 across all corruptions, a significant improvement over the Softmax Baseline (37.5) and IBE (40.3). Crucially, SCrIBE maintained a high mIOU on clean data (69.1 vs. 69.9 for the baseline), demonstrating that robustness does not require a trade-off in accuracy.

Qualitative Robustness

The visual impact of SCrIBE's robustness is most evident under severe corruption.

Figure 6: Qualitative comparison of segmentation masks under Gaussian noise, defocus blur, and snow. SCrIBE consistently produces more coherent masks than the baseline as corruption severity increases.

As shown in the figure above, while the baseline model's predictions fragment or disappear entirely under noise and blur, SCrIBE retains the general structure of the objects, demonstrating its superior ability to handle real-world sensor degradation.

Conclusion

Our research highlights that the way we structure class dependencies in neural networks profoundly affects their robustness. By moving away from the rigid competition of Softmax toward the more flexible, higher-dimensional representation of SCrIBE, we can build semantic segmentation models that are far better equipped for the unpredictable conditions of the real world.

Summary updated

BibTeX
@inproceedings{lehman2020structures,
  author = {Charles Lehman and Dogancan Temel and Ghassan Alregib},
  title = {{On the Structures of Representation for the Robustness of Semantic Segmentation to Input Corruption}},
  booktitle = {ICIP},
  year = {2020},
  doi = {10.1109/icip40778.2020.9190912},
}
Figure