Paracosm: A Test Framework for Autonomous Driving Simulations

Rupak Majumdar, Aman Shankar Mathur, Marcus Pirron, Laura Stegner, Damien ZuffereyFASE 2021 · DOI

Abstract

Testing autonomous vehicles (AVs) requires navigating a vast space of environmental conditions and dynamic interactions. We present Paracosm, a programmatic framework for stress-testing autonomous drivers through reactive simulations. By employing $k$-wise combinatorial testing for discrete variables and low-dispersion Halton sequences for continuous parameters, Paracosm systematically explores the scenario space to uncover edge-case failures. Evaluation across three neural network-based systems—road segmentation, pedestrian avoidance, and adaptive cruise control—demonstrates that Paracosm's systematic approach identifies significantly more failure modes than traditional random sampling.

Introduction

Testing autonomous vehicles in real-world environments is prohibitively expensive, slow, and dangerous. While virtual simulation offers a safer alternative, existing tools often suffer from a "customizability gap," relying on rigid environments or cumbersome GUI-based editors. Furthermore, traditional software testing metrics like code coverage are insufficient for the neural networks that drive modern AVs.

Paracosm addresses these challenges by providing a programmatic interface to define complex, reactive driving scenarios. It shifts the focus from code coverage to "world coverage," ensuring that the autonomous system is robust across a wide variety of environmental parameters.

System Architecture and Methodology

Paracosm is built on a synchronous reactive programming model using the Unity engine and UniRx. This architecture allows every element in the simulation—vehicles, pedestrians, and infrastructure—to be treated as a "reactive object" that consumes and produces data streams.

The Paracosm framework architecture, showing the interaction between the test generator, the system under test (SUT), and the reactive simulation environment.

The framework employs two primary strategies for parameter exploration:

  1. $k$-wise Combinatorial Testing: For discrete variables such as the number of lanes or car models, Paracosm ensures that every combination of $k$ parameters is tested at least once.
  2. Low-Dispersion Halton Sequences: For continuous variables like speed or fog density, the system uses quasi-Monte Carlo sequences. Unlike uniform random sampling, which can leave large gaps or create clusters, Halton sequences fill the parameter space evenly.

Systematic Parameter Exploration

By treating environmental factors as dynamic streams, Paracosm can simulate complex transitions, such as changing weather conditions or varying road layouts within a single test run.

Visualization of parameter streams (lanes, lighting, and camera views) used to define and sample simulation scenarios.

Experimental Evaluation

The authors evaluated Paracosm using three distinct Systems Under Test (SUTs):

  • Road Segmentation: A VGGNet CNN trained on dashcam imagery.
  • Jaywalking Pedestrian: A behavioral cloning CNN for collision avoidance.
  • Adaptive Cruise Control (ACC): A CNN trained for car-following behavior.

Key Results

The systematic exploration revealed significant performance degradations that random testing missed:

  • Failure Detection: In the jaywalking scenario, Halton sampling found 10% more failures than random sampling. When combined with local search (fuzzing parameters based on a cost function), the detection rate increased by 20%.
  • Environmental Sensitivity: The road segmentation model's accuracy dropped significantly (from 70% to 50%) when tested on configurations outside its training distribution.
  • Inactivity Failures: In ACC testing, the system identified that high fog density caused the vehicle to become overly "pessimistic," leading to failures where the car refused to move despite clear paths.

Conclusion and Future Work

Paracosm demonstrates that a programmatic, reactive approach to simulation can effectively identify edge-case failures in autonomous driving stacks. While challenges remain regarding the "sim-to-real" gap and the computational cost of high-fidelity simulations, the framework provides a rigorous foundation for measuring environmental coverage. Future work includes leveraging Paracosm for automated data augmentation, generating targeted training data to address specific model weaknesses identified during testing.

Summary updated

BibTeX
@inproceedings{majumdar2021paracosm,
  author = {Rupak Majumdar and Aman Shankar Mathur and Marcus Pirron and Laura Stegner and Damien Zufferey},
  title = {{Paracosm: A Test Framework for Autonomous Driving Simulations}},
  booktitle = {FASE},
  year = {2021},
  doi = {10.1007/978-3-030-71500-7\_9},
}
Figure