MIXSIM: A Hierarchical Framework for Mixed Reality Traffic Simulation

Simon Suo, Kelvin Wong, Justin Xu, James Tu, Alexander Cui, Sergio Casas, Raquel UrtasunCVPR 2023 · OpenReview

Abstract

The prevailing way to test a self-driving vehicle (SDV) in simulation involves non-reactive open-loop replay of real-world scenarios. However, to safely deploy SDVs, we must evaluate them in closed-loop environments. We propose MIXSIM, a hierarchical framework for mixed reality traffic simulation that leverages real-world recordings to create reactive and controllable "digital twins." By modeling agent goals as routes and learning a reactive route-conditional policy, MIXSIM enables testing different autonomy systems under identical conditions and exploring "what-if" situations with realistic variations or safety-critical interactions.

Introduction

The self-driving industry currently relies heavily on "open-loop" replay for testing. In this setup, a real-world recorded scenario is played back in a simulator. However, the other cars (agents) in the recording are "ghosts"—they follow their original paths regardless of what the SDV under test does. If the SDV brakes suddenly, the car behind it might phase right through it instead of stopping.

This lack of reactivity makes it impossible to evaluate "closed-loop" performance: how the SDV and the world interact over time. To safely deploy SDVs, we need to know how they handle "what-if" situations (e.g., "What if this driver was more aggressive?" or "What if I changed lanes now?"). Existing reactive models often suffer from a "sim-to-real" gap, producing robotic or unrealistic behaviors. MIXSIM addresses this by transforming static recordings into reactive digital twins.

MIXSIM transforms real-world scenarios into reactive digital twins, allowing for "what-if" testing of SDV and agent behaviors.

Methodology

MIXSIM introduces a hierarchical approach to bridge the gap between static recordings and reactive simulation by disentangling high-level goals from low-level maneuvers:

  1. High-Level Goals (Routes): The framework represents an agent's intent as a specific route through a lane graph. This structured representation is more flexible than a simple command but more constrained than a raw trajectory.
  2. Low-Level Policy: A reactive, route-conditional policy is learned from data. Given a route and the current state of surrounding traffic, this policy predicts the necessary acceleration and steering.
  3. The "Digital Twin" Workflow:
    • Route Inference: Using a Hidden Markov Model (HMM), MIXSIM back-calculates the most likely route an agent was taking in a recording.
  4. Reactive Re-simulation: Agents are controlled by the learned policy, following their intended routes while reacting (braking, swerving) to the SDV's actions.
  5. Controllability: Users can manually change an agent's route to create new scenarios, such as forcing a car to cut in front of the SDV.
The MIXSIM framework supports three main tasks: reactive re-simulation of real scenarios, sampling realistic variations, and finding safety-critical adversarial routes.

Key Results and Insights

The authors evaluated MIXSIM on the Argoverse 2 (AV2) urban dataset and a high-speed HIGHWAY simulation. MIXSIM significantly outperformed standard baselines like Behavior Cloning (BC) and pure Imitation Learning (IL):

  • Reconstruction Accuracy: On AV2, MIXSIM achieved a Final Displacement Error (FDE) of 5.04m, compared to 5.84m for IL and 14.48m for heuristic models.
  • Stability: It maintained the lowest collision rate (1.07%) and off-road rate (1.29%) during re-simulation, proving it creates a more stable environment.
  • Reactivity: In tests where the SDV behaved erratically, MIXSIM agents were the most successful at avoiding collisions (1.17 Collision Rate on Highway) compared to agents that blindly followed original paths.
  • Adversarial Testing: By using black-box optimization on agent routes, MIXSIM can automatically discover "safety-critical" scenarios, finding successful "attacks" on an SDV in fewer iterations (18.6) than trajectory-perturbation baselines.

Conclusion

MIXSIM demonstrates that disentangling goals from maneuvers is key to achieving high controllability without sacrificing realism. By using the lane graph as an inductive bias, the model naturally enforces traffic rules. While future work will focus on incorporating explicit traffic rules (like stop signs) and longitudinal diversity (e.g., aggressive vs. hesitant drivers), MIXSIM currently serves as a robust, reactive, and controllable digital twin for real-world autonomous vehicle testing.

Summary updated

Abstract

The prevailing way to test a self-driving vehicle (SDV) in simulation involves non-reactive open-loop replay of real world scenarios. However, in order to safely deploy SDVs to the real world, we need to evaluate them in closed-loop. Towards this goal, we propose to leverage the wealth of interesting scenarios captured in the real world and make them reactive and controllable to enable closed-loop SDV evaluation in what-if situations. In particular, we present MIXSIM, a hierarchical framework for mixed reality traffic simulation. MIXSIM explicitly models agent goals as routes along the road network and learns a reactive route-conditional policy. By inferring each agent's route from the original scenario, MIXSIM can reactively re-simulate the scenario and enable testing different autonomy systems under the same conditions. Furthermore, by varying each agent's route, we can expand the scope of testing to what-if situations with realistic variations in agent behaviors or even safety critical interactions. Our experiments show that MIXSIM can serve as a realistic, reactive, and controllable digital twin of real world scenarios. For more information, please visit the project website: https://waabi.ai/research/mixsim/
BibTeX
@inproceedings{suo2023mixsim,
  author = {Simon Suo and Kelvin Wong and Justin Xu and James Tu and Alexander Cui and Sergio Casas and Raquel Urtasun},
  title = {{MIXSIM: A Hierarchical Framework for Mixed Reality Traffic Simulation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=qlG0kYq8HN},
}
Figure