GAS: Generating Fast and Accurate Surrogate Models for Autonomous Vehicle Systems

Keyur Joshi, Chiao Hsieh, Sayan Mitra, Saša Misailovíc

This paper, presented by researchers from the University of Illinois Urbana-Champaign, introduces GAS (GPC for Autonomous Vehicle Systems). It is a framework designed to accelerate the safety and sensitivity analysis of autonomous vehicles (AVs) by replacing slow, high-fidelity simulations with fast, mathematically grounded "surrogate models."

TL;DR

GAS is the first framework to create accurate polynomial surrogate models for entire autonomous systems, including their complex neural-network-based perception. By abstracting perception into error distributions and applying Generalized Polynomial Chaos (GPC), it achieves an average 3.7x speedup in safety analysis while maintaining high fidelity to original simulations.

What Problem Does This Paper Address?

Evaluating the safety of autonomous vehicles is notoriously expensive. Developers typically rely on Monte Carlo Simulations (MCS)—running thousands of high-fidelity simulations (e.g., in Gazebo or CARLA) to see if a vehicle stays safe under various conditions.

The bottleneck is twofold:

  1. Computational Cost: Rendering images and running deep neural networks (DNNs) for perception in every simulation step is slow.
  2. Complexity for Surrogates: Traditional surrogate modeling techniques like Generalized Polynomial Chaos (GPC)—which approximate complex functions with simpler polynomials—struggle with AVs. This is because perception systems deal with high-dimensional "black-box" inputs (like raw pixels) and exhibit non-deterministic errors due to environmental factors (lighting, weather).

Methods

GAS overcomes these hurdles using a clever two-stage abstraction process:

  1. The Perception Model: Instead of trying to model how a neural network processes every possible pixel, GAS models the distribution of error. By sampling the perception system at various ground-truth states, GAS builds a polynomial regression model that predicts the mean and covariance of the perception error. This effectively turns a high-dimensional image-processing problem into a low-dimensional stochastic one.
  2. The Complete System Surrogate: GAS then composes this perception model with the vehicle’s control logic and physical dynamics. It uses GPC to create a single, unified polynomial that represents the entire closed-loop system.
  3. Handling Categorical Variables: Since GPC is designed for continuous variables, GAS uses Multi-Element GPC (ME-GPC) and ancillary classifiers to handle discrete states, such as different flight advisories in aircraft collision systems.

A key advantage of this method is compositionality: the perception model is independent of the controller. If an engineer tweaks the control algorithm, they can reuse the perception surrogate, saving hours of data collection.

Datasets

The authors evaluated GAS across five realistic scenarios:

  • Crop-Monitor: A robot navigating between crop rows (using ResNet-18).
  • Car-Straight & Car-Curved: Lane-keeping tasks (using LaneNet).
  • ACAS-Xu (Lookup Table & NN): Standard aircraft collision avoidance benchmarks.

The researchers used Gazebo 11 to generate photo-realistic training data, incorporating varied environmental factors like crop growth stages, tobacco vs. corn fields, and different lighting conditions.

Key Insights and Ideas

  • Modeling Distributions, Not Pixels: The "aha" moment is realizing that for safety analysis, we don't need to know what the camera sees; we need to know the statistical likelihood of the perception system being wrong.
  • Dimensionality Reduction via Abstraction: By replacing raw sensor data with an error distribution, the authors reduced the input space enough to make GPC—a technique usually reserved for simpler physical systems—viable for complex AVs.
  • Rapid Iteration: Because the perception model is decoupled, GAS enables a "plug-and-play" workflow. Developers can perform "what-if" analysis on control parameters in seconds rather than hours.

Key Results

  • Speed: GAS provided an average speedup of 3.7x (and up to 8.9x) for estimating the probability of a vehicle entering an unsafe state. For global sensitivity analysis, it was 1.4x faster.
  • Accuracy: The safe-state probabilities calculated by GAS matched the expensive Monte Carlo baselines for 97% or more of the simulation time steps.
  • Sensitivity Analysis: GAS successfully calculated Sobol indices (which identify which input variables most affect the output variance) with an average error of only 0.0004.

Limitations and Future Work

  • Environment Dependency: The perception surrogate is only as good as the environment distribution ($D_E$) it was trained on. If the vehicle encounters a scenario significantly different from the training set (e.g., snow when it was trained on sun), the surrogate accuracy will drop.
  • Discontinuities: As a polynomial-based method, GPC can struggle with systems that have sharp "jumps" or discontinuities in their logic, though the authors found that small time steps help mitigate this.
  • The Curse of Dimensionality: While GAS handles current benchmarks well, extremely high-dimensional state spaces (dozens of variables) still pose a challenge for GPC, potentially requiring further "truncation" techniques.

The authors plan to release GAS as an open-source tool, potentially transforming how AV developers handle regression testing and design exploration.

Summary updated

Abstract

Modern autonomous vehicle systems use complex perception and control components. These components can rapidly change during development of such systems, requiring constant re-testing. Unfortunately, high-fidelity simulations of these complex systems for evaluating vehicle safety are costly. The complexity also hinders the creation of less computationally intensive surrogate models. We present GAS, the first approach for creating surrogate models of complete (perception, control, and dynamics) autonomous vehicle systems containing complex perception and/or control components. GAS's two-stage approach first replaces complex perception components with a perception model. Then, GAS constructs a polynomial surrogate model of the complete vehicle system using Generalized Polynomial Chaos (GPC). We demonstrate the use of these surrogate models in two applications. First, we estimate the probability that the vehicle will enter an unsafe state over time. Second, we perform global sensitivity analysis of the vehicle system with respect to its state in a previous time step. GAS's approach also allows for reuse of the perception model when vehicle control and dynamics characteristics are altered during vehicle development, saving significant time. We consider five scenarios concerning crop management vehicles that must not crash into adjacent crops, self driving cars that must stay within their lane, and unmanned aircraft that must avoid collision. Each of the systems in these scenarios contain a complex perception or control component. Using GAS, we generate surrogate models for these systems, and evaluate the generated models in the applications described above. GAS's surrogate models provide an average speedup of $3.7\times$ for safe state probability estimation (minimum $2.1\times$) and $1.4\times$ for sensitivity analysis (minimum $1.3\times$), while still maintaining high accuracy.
BibTeX
@misc{joshi2022gas,
  author = {Keyur Joshi and Chiao Hsieh and Sayan Mitra and Saša Misailovíc},
  title = {{GAS: Generating Fast and Accurate Surrogate Models for Autonomous Vehicle Systems}},
  year = {2022},
  eprint = {2208.02232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.02232},
}
Figure