Abstract
Testing autonomous vehicles (AVs) requires navigating a vast space of environmental conditions and dynamic interactions. We present Paracosm, a programmatic framework for stress-testing autonomous drivers through reactive simulations. By employing $k$-wise combinatorial testing for discrete variables and low-dispersion Halton sequences for continuous parameters, Paracosm systematically explores the scenario space to uncover edge-case failures. Evaluation across three neural network-based systems—road segmentation, pedestrian avoidance, and adaptive cruise control—demonstrates that Paracosm's systematic approach identifies significantly more failure modes than traditional random sampling.
Introduction
Testing autonomous vehicles in real-world environments is prohibitively expensive, slow, and dangerous. While virtual simulation offers a safer alternative, existing tools often suffer from a "customizability gap," relying on rigid environments or cumbersome GUI-based editors. Furthermore, traditional software testing metrics like code coverage are insufficient for the neural networks that drive modern AVs.
Paracosm addresses these challenges by providing a programmatic interface to define complex, reactive driving scenarios. It shifts the focus from code coverage to "world coverage," ensuring that the autonomous system is robust across a wide variety of environmental parameters.
System Architecture and Methodology
Paracosm is built on a synchronous reactive programming model using the Unity engine and UniRx. This architecture allows every element in the simulation—vehicles, pedestrians, and infrastructure—to be treated as a "reactive object" that consumes and produces data streams.
The framework employs two primary strategies for parameter exploration:
- $k$-wise Combinatorial Testing: For discrete variables such as the number of lanes or car models, Paracosm ensures that every combination of $k$ parameters is tested at least once.
- Low-Dispersion Halton Sequences: For continuous variables like speed or fog density, the system uses quasi-Monte Carlo sequences. Unlike uniform random sampling, which can leave large gaps or create clusters, Halton sequences fill the parameter space evenly.
Systematic Parameter Exploration
By treating environmental factors as dynamic streams, Paracosm can simulate complex transitions, such as changing weather conditions or varying road layouts within a single test run.
Experimental Evaluation
The authors evaluated Paracosm using three distinct Systems Under Test (SUTs):
- Road Segmentation: A VGGNet CNN trained on dashcam imagery.
- Jaywalking Pedestrian: A behavioral cloning CNN for collision avoidance.
- Adaptive Cruise Control (ACC): A CNN trained for car-following behavior.
Key Results
The systematic exploration revealed significant performance degradations that random testing missed:
- Failure Detection: In the jaywalking scenario, Halton sampling found 10% more failures than random sampling. When combined with local search (fuzzing parameters based on a cost function), the detection rate increased by 20%.
- Environmental Sensitivity: The road segmentation model's accuracy dropped significantly (from 70% to 50%) when tested on configurations outside its training distribution.
- Inactivity Failures: In ACC testing, the system identified that high fog density caused the vehicle to become overly "pessimistic," leading to failures where the car refused to move despite clear paths.
Conclusion and Future Work
Paracosm demonstrates that a programmatic, reactive approach to simulation can effectively identify edge-case failures in autonomous driving stacks. While challenges remain regarding the "sim-to-real" gap and the computational cost of high-fidelity simulations, the framework provides a rigorous foundation for measuring environmental coverage. Future work includes leveraging Paracosm for automated data augmentation, generating targeted training data to address specific model weaknesses identified during testing.
Summary updated