TL;DR
Bench2Drive-Robust introduces the first device-centric benchmark to evaluate end-to-end autonomous driving models under realistic hardware and software deployment perturbations in a closed-loop simulator. By testing popular models against system-level imperfections like inference latency, camera frame drops, and GPS noise, the benchmark reveals critical control vulnerabilities that standard image-corruption evaluations completely miss.
What Problem Does This Paper Address?
End-to-end autonomous driving (E2E-AD) models have shown immense promise by mapping raw sensor inputs directly to driving trajectories or control commands. However, existing robustness benchmarks mostly focus on external appearance changes (such as adverse weather, rain, or lens blur) or evaluate models in an open-loop fashion (measuring prediction accuracy on pre-recorded logs without letting the model actually steer the vehicle).
In real-world deployment, autonomous vehicles face internal device defects and system-level constraints: compute delays, frozen camera streams, and noisy GPS signals. Because autonomous driving operates as a closed-loop feedback system, these minor execution and sensing errors can accumulate over time, leading to catastrophic control failures like lane deviations or collisions. This paper bridges the gap by introducing a benchmark specifically designed to stress-test E2E-AD models against these realistic deployment-side imperfections.
Methods
The authors present Bench2Drive-Robust, a modular framework that injects controlled, parameterized perturbations directly into the simulator-policy interface. Crucially, this is done non-invasively, meaning the underlying model architecture, checkpoints, and decision logic remain completely unchanged.
The benchmark categorizes deployment-relevant perturbations into three major families:
- Camera-Stream Failures: Models partial observation (applying black rectangular occlusion masks to random regions) and burst frame drops (where a camera freezes and repeatedly returns the most recent valid cached frame, simulating a stale but visually plausible stream rather than a black screen).
- Ego-State Estimation Errors: Models Gaussian GPS noise (which shifts the inferred navigation target) and speed noise (which systematically underestimates the vehicle's actual speed to simulate faulty speedometers or wheel slip).
- Compute-Induced Control Delay: Models inference latency using a First-In, First-Out (FIFO) action buffer. This delays command execution by fixed intervals (e.g., 100ms, 200ms, 500ms), ensuring reproducible testing regardless of the evaluation hardware's actual speed.
Datasets
The framework is built on top of Bench2Drive, a recent closed-loop evaluation benchmark utilizing the CARLA simulator. It evaluates driving policies across 220 routes spanning 44 interactive scenarios (such as navigating intersections, turning while yielding to pedestrians, and exiting parking bays) under various locations and weather conditions. Bench2Drive-Robust extends this setup by re-running these exact routes while systematically scaling up the severity of the injected deployment perturbations.
Key Insights and Ideas
- Clean Performance $\neq$ Robustness: A high score under ideal conditions does not guarantee safety during deployment. For instance, the model SimLingo achieved the highest clean driving score by a wide margin but proved highly vulnerable, degrading sharply under severe occlusion and latency.
- Latency Breaks Timing, Not Predictions: Compute delays do not necessarily cause models to output invalid paths or nonsensical waypoints. Instead, they cause otherwise reasonable actions to arrive too late for the evolved state of the environment, destabilizing the vehicle's motion.
- Auxiliary Sensors are Critical Failure Points: Even if the camera stream is perfectly clear, noise in non-visual inputs like GPS or speedometers can completely derail a vehicle. For example, GPS noise misaligns route tracking, causing the vehicle to miss turns, while speed underestimation leads to insufficient braking at red lights.
Key Results
The authors evaluated four representative E2E-AD models: TCP-traj, UniAD, VAD, and SimLingo.
- Inference Latency: All models degraded significantly as latency increased. SimLingo's Driving Score (DS) plummeted from a clean baseline of 85.94 down to 15.70 under a 500ms delay—a relative degradation of 82%.
- GPS Noise: TCP-traj proved highly sensitive to localization errors; its Driving Score dropped from 59.90 to 21.12 when subjected to severe GPS noise ($\sigma_{GPS} = 15\text{m}$), causing the vehicle to deviate from the road and become blocked.
- Camera Occlusion: Severe occlusion (mask ratio $r=0.8$) caused SimLingo's success rate to drop to 0.00%, demonstrating that spatial information loss severely impacts vision-centric models by hiding nearby actors and free space.
Limitations and Future Work
The primary limitation acknowledged by the authors is that Bench2Drive-Robust relies entirely on the CARLA simulator. Consequently, the simulation rendering and traffic behaviors may not perfectly mirror real-world driving distributions. Conversely, real-world dataset-based benchmarks lack the interactivity required for closed-loop evaluation.
The authors suggest that future work should focus on combining reactive simulation with real-world-calibrated perturbation statistics or leveraging generative simulation to improve both visual realism and interactive fidelity.
Summary updated