A simulation-based framework for functional testing of automated driving controllers

Adel Djoudi, Loïc Coquelin, Rémi Régnier

TL;DR

This paper introduces a novel simulation-based framework for rigorously testing the functional safety of automated driving controllers. By creating an "oracle" that defines the optimal "right decision" for a given scenario, the authors enable systematic evaluation of a controller's behavior against this ideal, paving the way for more robust and trustworthy autonomous systems.

What Problem Does This Paper Address?

Testing automated driving systems (ADS) presents a monumental challenge. The sheer number of possible driving scenarios, combined with the complexity of real-world environments and the need for high-fidelity sensor data, makes physical testing prohibitively expensive and time-consuming. It's practically impossible to accumulate the billions of miles required to cover all critical situations. Furthermore, ensuring the "functional safety" of these systems is paramount – they must not only perform their intended driving tasks but also react appropriately and safely to unexpected events or system failures. Existing safety standards, often based on traditional automotive systems, are insufficient for ADS, which require "fail-operational" capabilities rather than simply shutting down in a dangerous situation. This paper tackles the problem of how to rigorously and systematically evaluate the decision-making logic of ADS controllers in a controlled, repeatable, and cost-effective manner.

Methods

The core of the proposed solution is an optimization-based reference behavior model, which acts as a "test oracle." This oracle takes a given scenario (including future information about the environment, such as the position of other vehicles) and computes the "right decision" – specifically, the optimal acceleration and steering commands – that an ideal ADS controller should execute. This reference behavior is computed offline, leveraging the omniscient nature of simulation to know future states.

The framework then employs a three-step testing architecture:

  1. ADS Simulation: The scenario is implemented in a simulator (SCANeR Studio in this case), and the ADS controller under test receives simulated sensor data and computes its own acceleration and steering commands.
  2. Reference Computation: The optimization-based oracle, using the same scenario data, computes the ideal reference commands.
  3. Test Verdict: The actual commands from the ADS controller are compared against the reference commands. This comparison, along with an analysis of distances to other vehicles and adherence to safety constraints (like deceleration limits), forms the basis for evaluating the ADS's performance and classifying the criticality of the scenario.

The optimization problem aims to minimize the difference between the current distance to a lead vehicle and a desired regulation distance (defined by a Time-to-Vehicle, TIV, parameter), while also penalizing excessive acceleration, jerk, and relative speed. This ensures the reference behavior is not only safe but also comfortable.

Datasets

The paper does not explicitly mention or use pre-existing datasets for training or evaluation in the traditional machine learning sense. Instead, it focuses on generating and utilizing scenarios within a simulation environment. The authors use the SCANeR Studio simulator, which allows for the creation and variation of scenarios. For their case study, they specifically focus on a "cut-in" scenario, where a new vehicle enters the lane in front of the ego vehicle. They generate 30 test cases for this scenario by varying parameters like vehicle speeds and the timing of the cut-in.

Key Insights and Ideas

  • The "Oracle" Approach: Defining an "ideal" or "right" decision for a given situation is a powerful way to create objective benchmarks for ADS testing. This moves beyond simply checking for collisions to evaluating the quality and safety of the driving behavior.
  • Offline Reference for Online Testing: Leveraging simulation's ability to see the future allows for the offline computation of a perfect reference behavior. This reference can then be used to evaluate the real-time performance of the ADS controller.
  • Systematic Scenario Classification: The framework enables a systematic classification of scenarios based on their criticality, allowing developers to prioritize testing efforts and understand the limitations of their ADS.
  • Bridging Functionality and Safety: The paper emphasizes that functional safety cannot be separated from the intended functionality. The reference model is designed to satisfy both comfort and safety requirements, demonstrating how these aspects are intertwined.

Key Results

The paper demonstrates the framework's utility through a case study of an Adaptive Cruise Control (ACC) controller in a "cut-in" scenario.

  • Quantitative Comparison: Table I shows the results of 30 test cases, comparing the minimum and mean distances to the lead vehicle achieved by the Ego vehicle (controlled by the ADS) against reference behaviors calculated for TIVs of 1s, 2s, and 3s.
  • Behavioral Insights: The results indicate that the Ego vehicle's behavior is generally close to the reference with a TIV of 2s, suggesting the controller is parameterized accordingly. The computed references are shown to be conservative, maintaining a larger distance than the Ego vehicle in some cases.
  • Criticality Classification: Table II illustrates how scenarios can be classified into "low," "medium," and "high" criticality levels based on the comparison with different reference TIVs. This provides a valuable tool for understanding risk.
  • Framework Validation: The implementation within SCANeR Studio demonstrates the feasibility of the proposed testing architecture, integrating environment simulation, reference computation, and analysis.

Limitations and Future Work

The authors acknowledge several areas for future development:

  • More Realistic Vehicle Models: The current work uses a simplified double integrator model for the vehicle dynamics. Future work will involve using more sophisticated and realistic vehicle models in the reference computation.
  • Automated Reference Model Synthesis: The current process of defining the reference behavior is manual. The authors plan to develop methods for automatically synthesizing ADS controller reference models, which would significantly scale up the testing process.
  • Wider Scenario Databases: The framework will be integrated with larger databases of scenarios to retrieve a wider range of input test data, further enhancing its applicability.
  • Abstraction of Safety Requirements: Developing appropriate abstraction levels to describe safety requirements and goals for a wider range of scenarios is an ongoing challenge.

Summary updated

Abstract

Article published in proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (September 20 – 23, 2020)
BibTeX
@inproceedings{djoudi2020simulation,
  author = {Adel Djoudi and Loïc Coquelin and Rémi Régnier},
  title = {{A simulation-based framework for functional testing of automated driving controllers}},
  booktitle = {HAL},
  year = {2020},
}
Figure