S-RAF: A Simulation-Based Robustness Assessment Framework for Responsible Autonomous Driving

Daniel Omeiza, Pratik Somaiya, Jo-Ann Pattinson, Carolyn Ten Holter, Jack Stilgoe, Marina Jirotka, Lars Kunze

Abstract

As autonomous driving (AD) technology transitions from controlled research environments to public infrastructure, the need for standardized "Responsible AI" (RAI) metrics becomes critical. This paper introduces the Simulation-Based Robustness Assessment Framework (S-RAF), a benchmarking tool designed to evaluate AD agents across three dimensions: robustness to environmental disturbances, environmental sustainability, and process transparency. Leveraging the CARLA simulator, S-RAF stress-tests state-of-the-art models against sensor failures and edge cases while quantifying the carbon footprint of their computational requirements. Our findings highlight a significant trade-off between model robustness and energy efficiency, providing a foundation for more holistic AD auditing.

Introduction

The development of Autonomous Driving (AD) systems has historically prioritized performance metrics such as route completion and collision avoidance in ideal conditions. However, this narrow focus ignores the complexities of real-world deployment, where sensor degradation and environmental unpredictability are common. Furthermore, the industry lacks a unified standard for Responsible AI (RAI), often neglecting the environmental cost of the high-compute models required for safe navigation. S-RAF addresses these gaps by providing a transparent, objective framework to assess robustness and sustainability before vehicles are deployed on public roads.

The S-RAF Framework

S-RAF is built upon the CARLA Driving Simulator and introduces a rigorous testing methodology focused on three primary pillars:

1. Robustness Assessment

The framework injects various "disturbances" to evaluate how agents handle sub-optimal conditions:

  • Environmental Disturbances: Simulations of camera and LiDAR occlusion (e.g., dirt, snow, or debris) alongside adverse weather conditions like heavy fog and torrential rain.
  • Sensor Errors: Injection of "salt and pepper" noise into visual feeds and hardware-level failure simulations, such as the loss of specific LiDAR channels.
  • Corner Cases: Introduction of unpredictable actor behaviors, including jaywalkers and sudden traffic shifts, to test the agent's resilience to data drift.

2. Sustainability and Carbon Tracking

To address the "Sustainability Gap," S-RAF utilizes the codecarbon library to track GPU and RAM energy consumption during inference. This data is converted into CO2 equivalents based on local energy grid intensity, allowing researchers to quantify the environmental cost of every mile driven.

3. Transparency Auditing

S-RAF includes a qualitative "process-based transparency" check. This audit evaluates whether models provide open-source code, pre-trained weights, and comprehensive documentation, such as Model Cards or Datasheets, to ensure accountability.

Methodology and Metrics

The framework introduces the Robustness Driving Score (RDS), which calculates the ratio of an agent's performance under disturbed conditions relative to its performance in a "clean" environment. This allows for the identification of "brittle" models that may perform well in sunshine but fail under minor sensor stress.

For evaluation, the authors benchmarked three prominent AD agents:

  1. LBC (2020): Learning by Cheating.
  2. NEAT (2021): Neural Attention Fields.
  3. InterFuser (2022): A multi-modal sensor fusion transformer.

A non-machine-learning CARLA NPC Agent served as a baseline for "ideal" carbon emissions, as it operates on ground-truth data without the need for heavy deep-learning inference.

Key Insights and Results

The Robustness-Sustainability Trade-off

Experimental results revealed a clear scaling law: increased robustness comes at a high environmental price. InterFuser emerged as the most resilient model, achieving a Regular Driving Score of 54.36% and a Robustness Driving Score of 34.55%. However, it emitted 0.0341 kg CO2/sec—nearly 2.5 times the emissions of the LBC model. This suggests that as we add complex sensor fusion layers to improve safety, we significantly increase the carbon footprint of the vehicle.

Vulnerability to Visual Noise

Across all tested models, camera-based noise and occlusion caused the most significant performance degradation. This finding is particularly relevant given the industry trend toward "camera-only" navigation systems, which may lack the redundancy needed to handle lens contamination or extreme weather.

Performance vs. Safety

The study demonstrated that high performance in standard benchmarks does not guarantee safety. Some models exhibited high regular driving scores but low RDS, indicating they are highly optimized for specific conditions but lack the resilience required for real-world unpredictability.

Limitations and Future Work

While S-RAF provides a vital auditing tool, it is currently limited by the fidelity of synthetic data compared to real-world complexity. Additionally, the carbon indicator currently only accounts for inference; future iterations will aim to include the CO2 costs of model training and hardware manufacturing. The authors intend to expand S-RAF to include fairness and explainability metrics, ultimately hosting a public "RAI Leaderboard" to encourage the development of safer, greener autonomous systems.

Conclusion

S-RAF shifts the focus of AD evaluation from pure performance to a multi-dimensional assessment of responsibility. By highlighting the trade-offs between safety and sustainability, the framework provides a roadmap for developers and regulators to build autonomous systems that are not only robust in the face of adversity but also mindful of their environmental impact.

Summary updated

BibTeX
@inproceedings{omeiza2024s,
  author = {Daniel Omeiza and Pratik Somaiya and Jo-Ann Pattinson and Carolyn Ten Holter and Jack Stilgoe and Marina Jirotka and Lars Kunze},
  title = {{S-RAF: A Simulation-Based Robustness Assessment Framework for Responsible Autonomous Driving}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v4i1.31776},
}
Figure