Abstract
Panoptic segmentation—the joint task of semantic and instance segmentation—is vital for the situational awareness of Assisted and Automated Driving (AAD) systems. While current models achieve high performance on clean datasets, their robustness against real-world camera degradations remains a critical concern. This study introduces a systematic benchmark for evaluating panoptic segmentation robustness. By injecting 19 types of physically-grounded degradations into the Cityscapes dataset, we evaluate various architectures and find that Vision Transformer (ViT)-based models significantly outperform traditional CNNs. Our analysis identifies specific "performance killers" and establishes a strong correlation between perceptual image quality metrics and segmentation performance.
Introduction
For AAD systems, panoptic segmentation provides a comprehensive pixel-level understanding of the environment, identifying both background categories ("stuff") and individual objects ("things"). However, most research relies on datasets captured under ideal conditions. In practice, automotive cameras encounter diverse degradations, including adverse weather, sensor noise, and lens obstructions. Understanding how these factors impact Panoptic Quality (PQ) is essential for ensuring the safety and reliability of autonomous vehicles.
Methodology
The authors developed a unifying pipeline to bridge the gap between image quality and perception performance. This pipeline involves a taxonomy of 19 degradation factors categorized into environmental conditions, optic obstructions, Image Signal Processor (ISP) failures, and sensor/motion noise.
Unlike previous approaches using simple image editing, this work employs physically-grounded models. For instance, "unfavourable light" (night and extreme low-light) is simulated using CycleGAN and Zero-DCE, while "snow" incorporates a "veiling effect" based on light scattering theories. The benchmark evaluates three prominent architectures: Panoptic DeepLab (CNN), EfficientPS, and OneFormer (ViT-based).
The D-Cityscapes+ Benchmark
To facilitate robustness testing, the authors introduced D-Cityscapes+ (Degraded-Cityscapes plus). This benchmark applies the 19 identified degradation types across three severity levels to the 500 validation images of the Cityscapes dataset, creating a diverse, paired dataset for robustness testing.
Key Insights and Results
The benchmarking process yielded several critical findings:
- The ViT Advantage: Models with Vision Transformer backbones (e.g., OneFormer with ConvNeXt-L) proved significantly more robust than CNN-based counterparts (e.g., Panoptic DeepLab with ResNet). ViTs' global receptive fields likely allow them to maintain context despite local noise.
- Performance Killers: Droplets on the lens and Gaussian noise were found to be the most detrimental. Under the highest severity of Gaussian noise, PQ dropped from a clean score of ~68.5 down to a staggering 11.5.
- Predictive Metrics: While traditional metrics like PSNR are poor indicators of model performance, Learned Perceptual Image Patch Similarity (LPIPS) and CW-SSIM showed a strong correlation with PQ drops, suggesting their utility as real-time "vision health" monitors.
- Weather Hierarchy: The models generally found snow to be the most challenging condition, followed by rain and fog (Snow > Rain > Fog). The "veiling effect" in the new snow model proved particularly challenging for instance boundary detection.
Conclusion
This research highlights the vulnerability of current panoptic segmentation models to real-world camera degradations. The superior performance of ViT-based architectures and the identification of LPIPS as a reliable failure predictor provide a roadmap for developing more robust AAD perception systems. Future work will focus on using the D-Cityscapes+ dataset for robust training and validating these findings with real-world sensor data captured in extreme environments.
Summary updated