Benchmarking the Robustness of Semantic Segmentation Models

Christoph Kamann, Carsten Rother

Introduction

Deep Convolutional Neural Networks (DCNNs) have become the standard for semantic segmentation, the task of assigning a class label to every pixel in an image. While these models achieve high accuracy on clean, curated datasets, they often exhibit significant "brittleness" when deployed in real-world environments. Factors such as sensor noise, motion blur, and adverse weather conditions can cause performance to plummet, posing serious risks for safety-critical applications like autonomous driving. This study provides the first exhaustive robustness benchmark for semantic segmentation, evaluating how state-of-the-art models handle 19 different types of image corruptions.

Methodology

The researchers utilized the DeepLabv3+ architecture as a primary testbed, conducting an extensive ablation study to isolate the impact of specific architectural components. Key elements tested included:

  • Atrous (Dilated) Convolutions: Filters that expand the field of view without increasing parameter count.
  • Atrous Spatial Pyramid Pooling (ASPP): A module designed to capture multi-scale context.
  • Dense Prediction Cell (DPC): A high-performance module discovered via Neural Architecture Search (NAS).
  • Long-Range Links: Skip connections that preserve spatial detail from early layers.

The models were benchmarked using nearly 400,000 images derived from the Cityscapes, PASCAL VOC 2012, and ADE20K datasets. Corruptions included the standard ImageNet-C set (blur, noise, weather) as well as custom "realistic" corruptions like intensity-dependent camera noise and lens aberrations (PSF blur).

Architectural Insights: The DPC Paradox

A significant finding of this research is the inverse relationship between certain performance-boosting modules and model robustness. Specifically, the Dense Prediction Cell (DPC), while effective at increasing mIoU on clean data, was found to significantly degrade robustness. The DPC appears to overfit to clean signals, making the model highly vulnerable to even moderate levels of noise.

Figure 1: Segmentation results with the Dense Prediction Cell (DPC) under image corruption, showing significant noise and artifacts in the prediction mask.
Figure 2: Segmentation results without the DPC module. The model demonstrates superior robustness and cleaner predictions under identical corruption levels.

In contrast, Atrous convolutions were found to naturally enhance robustness. By sampling pixels at a distance, these layers avoid overfitting to local pixel-level noise, allowing the model to rely more on broader spatial context.

Key Results and Discussion

Unlike image classification, where higher accuracy does not always imply greater robustness, this study found that better-performing segmentation models are generally more resilient to corruptions.

  • Backbone Performance: Xception-71 emerged as the most robust backbone for DeepLabv3+ across nearly all categories.
  • Corruption Sensitivity: Models were particularly sensitive to corruptions that destroy texture, such as Gaussian noise and JPEG compression, which could reduce mIoU by over 50%. Conversely, they remained relatively stable under geometric distortions and brightness changes.
  • The GSCNN Paradox: The Gated-ShapeCNN (GSCNN), which achieves top-tier results on clean benchmarks, proved to be significantly more brittle than Xception-based models when faced with sensor noise.

Conclusion

This benchmarking effort highlights a critical gap between "clean-room" performance and real-world reliability. The findings suggest that future Neural Architecture Search (NAS) should incorporate corrupted data into the search process to ensure that discovered modules, like the DPC, do not sacrifice robustness for marginal gains in clean accuracy. Moving forward, the goal is to develop models that are not just accurate, but dependable in the unpredictable conditions of the real world.

Summary updated

BibTeX
@inproceedings{kamann2020benchmarking,
  author = {Christoph Kamann and Carsten Rother},
  title = {{Benchmarking the Robustness of Semantic Segmentation Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  doi = {10.1109/cvpr42600.2020.00885},
}
Figure