Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Mark Koren, Mykel J. Kochenderfer

Abstract

Safety validation for autonomous vehicles (AVs) requires identifying rare but critical failure scenarios. Adaptive Stress Testing (AST) provides a framework for finding these scenarios using reinforcement learning, but traditional methods often require internal simulator access and fail to generalize across varying initial conditions. This paper introduces a Generalized Recurrent Deep Reinforcement Learning (GRDRL) solver that treats the simulator as a black box and generalizes across a continuous space of scenarios. Experimental results demonstrate that this approach is more reliable, finds more likely failures, and offers a massive computational speedup compared to existing baselines.

Introduction

To ensure the safety of autonomous systems, engineers must test them against dangerous edge cases. Since these events are rare in the real world, simulation is the primary tool for validation. Adaptive Stress Testing (AST) frames the search for these failures as a Markov Decision Process (MDP), where an adversarial agent controls the environment to induce a crash in the system under test.

However, existing Deep Reinforcement Learning (DRL) solvers for AST face two significant hurdles. First, they typically require "white-box" access to the simulator's internal state, which is often unavailable in complex third-party software. Second, they suffer from the "curse of discretization," requiring a separate solver for every specific starting condition, making large-scale validation computationally prohibitive.

Methods

The proposed solution addresses these limitations by utilizing a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) layers.

Black-Box Independence

By using an RNN, the solver can maintain a memory of past actions. Since the system under test is deterministic relative to environmental actions, this action history serves as a sufficient proxy for the simulator's internal state. This allows the solver to operate effectively without needing to peek inside the simulator, fulfilling the "black-box" requirement.

Continuous Scenario Generalization

To enable generalization, the authors developed an architecture where the initial state is concatenated with the previous action as input to the network. During training, initial conditions are sampled from a continuous uniform distribution. This allows a single GRDRL solver to learn a robust policy capable of finding failures from any starting point within the defined space.

The generalized solver architecture allows a single model to handle multiple initial conditions, enabling continuous scenario generalization.

Key Results

The GRDRL solver was evaluated in a simulated neighborhood environment where a pedestrian crosses in front of an AV. The approach was compared against a discrete RNN (DRDRL), a standard MLP solver, and a Monte Carlo Tree Search (MCTS) baseline.

  • Reliability: The GRDRL solver found collisions in 100% of the 32 test bins, significantly outperforming MCTS (65.63%) and the MLP solver (90.63%).
  • Failure Likelihood: The generalized solver identified more "likely" failures, achieving a maximum collision reward of -91.67. In this context, rewards represent the Mahalanobis distance of actions; a higher (less negative) value indicates a failure that requires less extreme or unlikely environmental behavior.
  • Efficiency: The generalized architecture demonstrated superior sample efficiency, converging in approximately 100 iterations, whereas discrete methods failed to reach similar performance even after 500 iterations.
The generalized solver (black line) converges significantly faster and finds more likely failures (higher Mahalanobis distance) than discrete baselines.

Conclusion

This research demonstrates that using recurrent architectures in AST removes the need for internal simulator state access and enables efficient generalization across continuous scenario spaces. Not only does this make the validation process more computationally tractable, but it also improves safety by identifying critical vulnerabilities that discrete grid-search methods might overlook. Future work will explore the scalability of this approach in high-fidelity, high-dimensional autonomous driving simulations.

Summary updated

BibTeX
@inproceedings{koren2019efficient,
  author = {Mark Koren and Mykel J. Kochenderfer},
  title = {{Efficient Autonomy Validation in Simulation with Adaptive Stress Testing}},
  booktitle = {ITSC},
  year = {2019},
  doi = {10.1109/itsc.2019.8917403},
}
Figure