Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations

Zhiyuan Zhang, Zhenghao Jin, Yanlun Peng, Xianda Guo, Haoran Liu, Shaofeng Zhang, Xingjun Ma, Zuxuan Wu, Junchi Yan, Xiaosong Jia, Yu-Gang JiangarXiv 2026 · arXiv

TL;DR

Bench2Drive-Robust introduces the first device-centric benchmark to evaluate end-to-end autonomous driving models under realistic hardware and software deployment perturbations in a closed-loop simulator. By testing popular models against system-level imperfections like inference latency, camera frame drops, and GPS noise, the benchmark reveals critical control vulnerabilities that standard image-corruption evaluations completely miss.

What Problem Does This Paper Address?

End-to-end autonomous driving (E2E-AD) models have shown immense promise by mapping raw sensor inputs directly to driving trajectories or control commands. However, existing robustness benchmarks mostly focus on external appearance changes (such as adverse weather, rain, or lens blur) or evaluate models in an open-loop fashion (measuring prediction accuracy on pre-recorded logs without letting the model actually steer the vehicle).

In real-world deployment, autonomous vehicles face internal device defects and system-level constraints: compute delays, frozen camera streams, and noisy GPS signals. Because autonomous driving operates as a closed-loop feedback system, these minor execution and sensing errors can accumulate over time, leading to catastrophic control failures like lane deviations or collisions. This paper bridges the gap by introducing a benchmark specifically designed to stress-test E2E-AD models against these realistic deployment-side imperfections.

Figure 2: Overview of Bench2Drive-Robust. We evaluate end-to-end autonomous driving models in closed-loop simulation by injecting deployment-relevant perturbations into the sensing, ego-state, and action pipelines while keeping the evaluated policy unchanged. The benchmark covers temporal delays, observation integrity perturbations, and ego-state estimation errors, enabling controlled robustness evaluation under consistent driving scenarios.

Methods

The authors present Bench2Drive-Robust, a modular framework that injects controlled, parameterized perturbations directly into the simulator-policy interface. Crucially, this is done non-invasively, meaning the underlying model architecture, checkpoints, and decision logic remain completely unchanged.

The benchmark categorizes deployment-relevant perturbations into three major families:

  1. Camera-Stream Failures: Models partial observation (applying black rectangular occlusion masks to random regions) and burst frame drops (where a camera freezes and repeatedly returns the most recent valid cached frame, simulating a stale but visually plausible stream rather than a black screen).
  2. Ego-State Estimation Errors: Models Gaussian GPS noise (which shifts the inferred navigation target) and speed noise (which systematically underestimates the vehicle's actual speed to simulate faulty speedometers or wheel slip).
  3. Compute-Induced Control Delay: Models inference latency using a First-In, First-Out (FIFO) action buffer. This delays command execution by fixed intervals (e.g., 100ms, 200ms, 500ms), ensuring reproducible testing regardless of the evaluation hardware's actual speed.

Datasets

The framework is built on top of Bench2Drive, a recent closed-loop evaluation benchmark utilizing the CARLA simulator. It evaluates driving policies across 220 routes spanning 44 interactive scenarios (such as navigating intersections, turning while yielding to pedestrians, and exiting parking bays) under various locations and weather conditions. Bench2Drive-Robust extends this setup by re-running these exact routes while systematically scaling up the severity of the injected deployment perturbations.

Key Insights and Ideas

  • Clean Performance $\neq$ Robustness: A high score under ideal conditions does not guarantee safety during deployment. For instance, the model SimLingo achieved the highest clean driving score by a wide margin but proved highly vulnerable, degrading sharply under severe occlusion and latency.
  • Latency Breaks Timing, Not Predictions: Compute delays do not necessarily cause models to output invalid paths or nonsensical waypoints. Instead, they cause otherwise reasonable actions to arrive too late for the evolved state of the environment, destabilizing the vehicle's motion.
  • Auxiliary Sensors are Critical Failure Points: Even if the camera stream is perfectly clear, noise in non-visual inputs like GPS or speedometers can completely derail a vehicle. For example, GPS noise misaligns route tracking, causing the vehicle to miss turns, while speed underestimation leads to insufficient braking at red lights.

Key Results

The authors evaluated four representative E2E-AD models: TCP-traj, UniAD, VAD, and SimLingo.

  • Inference Latency: All models degraded significantly as latency increased. SimLingo's Driving Score (DS) plummeted from a clean baseline of 85.94 down to 15.70 under a 500ms delay—a relative degradation of 82%.
  • GPS Noise: TCP-traj proved highly sensitive to localization errors; its Driving Score dropped from 59.90 to 21.12 when subjected to severe GPS noise ($\sigma_{GPS} = 15\text{m}$), causing the vehicle to deviate from the road and become blocked.
  • Camera Occlusion: Severe occlusion (mask ratio $r=0.8$) caused SimLingo's success rate to drop to 0.00%, demonstrating that spatial information loss severely impacts vision-centric models by hiding nearby actors and free space.

Limitations and Future Work

The primary limitation acknowledged by the authors is that Bench2Drive-Robust relies entirely on the CARLA simulator. Consequently, the simulation rendering and traffic behaviors may not perfectly mirror real-world driving distributions. Conversely, real-world dataset-based benchmarks lack the interactivity required for closed-loop evaluation.

The authors suggest that future work should focus on combining reactive simulation with real-world-calibrated perturbation statistics or leveraging generative simulation to improve both visual realism and interactive fidelity.

Summary updated

Abstract

Robustness is a critical requirement for deploying autonomous driving systems in the real world. Existing robustness benchmarks for autonomous driving have made important progress in studying the effects of image-level corruptions, such as adverse weather or camera degradation, on perception modules and open-loop planning outputs. However, deployment can also involve system-level imperfections, such as inference latency and ego-state estimation errors, which remain less studied in closed-loop E2E-AD evaluation. These imperfections can accumulate through the feedback loop and destabilize control. In this work, we present Bench2Drive-Robust, to our knowledge the first device-centric robustness benchmark for closed-loop end-to-end autonomous driving under realistic deployment perturbations. We systematically evaluate deployment-oriented perturbations arising from three major sources: camera-stream failures (frame drop, partial observation), ego-state estimation errors (GPS noise, and speed or odometry errors), and compute-induced control delay (model inference delay). We evaluate representative end-to-end driving methods and analyze their robustness under different perturbation severities. Our results show that these deployment-related perturbations can substantially degrade closed-loop driving performance, revealing robustness challenges that are not fully captured by conventional image-level corruption evaluations. By establishing a closed-loop evaluation protocol and demonstrating the substantial impact of these deployment-oriented perturbations, Bench2Drive-Robust defines practical robustness problems for end-to-end autonomous driving and encourages further research on deployment-aware robust driving systems.
BibTeX
@misc{zhang2026bench,
  author = {Zhiyuan Zhang and Zhenghao Jin and Yanlun Peng and Xianda Guo and Haoran Liu and Shaofeng Zhang and Xingjun Ma and Zuxuan Wu and Junchi Yan and Xiaosong Jia and Yu-Gang Jiang},
  title = {{Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations}},
  year = {2026},
  eprint = {2605.18059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18059},
}

Loading related content...

Figure