Abstract
The prevailing way to test a self-driving vehicle (SDV) in simulation involves non-reactive open-loop replay of real-world scenarios. However, to safely deploy SDVs, we must evaluate them in closed-loop environments. We propose MIXSIM, a hierarchical framework for mixed reality traffic simulation that leverages real-world recordings to create reactive and controllable "digital twins." By modeling agent goals as routes and learning a reactive route-conditional policy, MIXSIM enables testing different autonomy systems under identical conditions and exploring "what-if" situations with realistic variations or safety-critical interactions.
Introduction
The self-driving industry currently relies heavily on "open-loop" replay for testing. In this setup, a real-world recorded scenario is played back in a simulator. However, the other cars (agents) in the recording are "ghosts"—they follow their original paths regardless of what the SDV under test does. If the SDV brakes suddenly, the car behind it might phase right through it instead of stopping.
This lack of reactivity makes it impossible to evaluate "closed-loop" performance: how the SDV and the world interact over time. To safely deploy SDVs, we need to know how they handle "what-if" situations (e.g., "What if this driver was more aggressive?" or "What if I changed lanes now?"). Existing reactive models often suffer from a "sim-to-real" gap, producing robotic or unrealistic behaviors. MIXSIM addresses this by transforming static recordings into reactive digital twins.
Methodology
MIXSIM introduces a hierarchical approach to bridge the gap between static recordings and reactive simulation by disentangling high-level goals from low-level maneuvers:
- High-Level Goals (Routes): The framework represents an agent's intent as a specific route through a lane graph. This structured representation is more flexible than a simple command but more constrained than a raw trajectory.
- Low-Level Policy: A reactive, route-conditional policy is learned from data. Given a route and the current state of surrounding traffic, this policy predicts the necessary acceleration and steering.
- The "Digital Twin" Workflow:
- Route Inference: Using a Hidden Markov Model (HMM), MIXSIM back-calculates the most likely route an agent was taking in a recording.
- Reactive Re-simulation: Agents are controlled by the learned policy, following their intended routes while reacting (braking, swerving) to the SDV's actions.
- Controllability: Users can manually change an agent's route to create new scenarios, such as forcing a car to cut in front of the SDV.
Key Results and Insights
The authors evaluated MIXSIM on the Argoverse 2 (AV2) urban dataset and a high-speed HIGHWAY simulation. MIXSIM significantly outperformed standard baselines like Behavior Cloning (BC) and pure Imitation Learning (IL):
- Reconstruction Accuracy: On AV2, MIXSIM achieved a Final Displacement Error (FDE) of 5.04m, compared to 5.84m for IL and 14.48m for heuristic models.
- Stability: It maintained the lowest collision rate (1.07%) and off-road rate (1.29%) during re-simulation, proving it creates a more stable environment.
- Reactivity: In tests where the SDV behaved erratically, MIXSIM agents were the most successful at avoiding collisions (1.17 Collision Rate on Highway) compared to agents that blindly followed original paths.
- Adversarial Testing: By using black-box optimization on agent routes, MIXSIM can automatically discover "safety-critical" scenarios, finding successful "attacks" on an SDV in fewer iterations (18.6) than trajectory-perturbation baselines.
Conclusion
MIXSIM demonstrates that disentangling goals from maneuvers is key to achieving high controllability without sacrificing realism. By using the lane graph as an inductive bias, the model naturally enforces traffic rules. While future work will focus on incorporating explicit traffic rules (like stop signs) and longitudinal diversity (e.g., aggressive vs. hesitant drivers), MIXSIM currently serves as a robust, reactive, and controllable digital twin for real-world autonomous vehicle testing.
Summary updated