ForSim: Stepwise Forward Simulation for Traffic Policy Fine-Tuning

Keyu Chen, Wenchao Sun, Hao Cheng, Zheng Fu, Sifa Zheng

TL;DR

ForSim is a new stepwise, closed-loop forward simulation paradigm designed to make traffic simulators more realistic and reactive. By replacing static, open-loop rollouts with a method that dynamically aligns trajectories and updates agent predictions at every timestep, ForSim significantly reduces collisions and improves the physical plausibility of simulated driving behaviors.

What Problem Does This Paper Address?

Training autonomous vehicles (AVs) requires high-fidelity traffic simulators that can mimic the messy, interactive nature of real-world driving. However, current simulators struggle with two main issues:

  1. Covariate Shift: When a model is trained on "expert" data (imitation learning) but then deployed in a simulator, small errors accumulate. Eventually, the agent finds itself in a state it never saw during training, leading to catastrophic failure.
  2. Multimodality vs. Realism: Real drivers have many ways to handle a situation (e.g., overtaking on the left vs. right). Existing "closed-loop" fine-tuning methods often suffer from mode collapse, where the model forgets these diverse behaviors and converges on a single, boring "average" behavior.

Recent frameworks like RIFT tried to solve this but used "non-reactive" simulations—meaning that during the "imagined" future steps of a rollout, other cars didn't react to the ego-vehicle's actions. This lack of interaction makes the simulation's feedback loop unrealistic.

Methods

The authors introduce ForSim, which decomposes forward simulation into two interactive components:

  • Trajectory-Aligned Rollout (for the Traffic Agent): Instead of just following a single pre-planned path, the agent generates multiple candidate trajectories. At every virtual timestep, it selects the candidate that best matches a fixed "reference" trajectory spatiotemporally. This selection is then executed through a PID controller and a kinematic bicycle model. This ensures the agent stays "true" to its intended behavior (preserving multimodality) while remaining physically grounded.
  • Stepwise Prediction Rollout (for Other Agents): In most simulators, "other" cars just follow a fixed path or a constant velocity. In ForSim, a prediction model re-forecasts the paths of all other agents at every virtual step based on the new state. This creates a reactive environment where agents "respond" to each other’s movements.

These components are integrated into the RIFT framework, using Group-Relative Policy Optimization to fine-tune the driving policy based on the rewards (safety, efficiency, comfort) earned during these realistic rollouts.

Datasets

The experiments were conducted using the CARLA simulator, an industry-standard open-source platform for autonomous driving research.

  • The authors specifically utilized the PDM-Lite AV setting for evaluation.
  • They compared ForSim against a wide array of baselines, including Pluto (Imitation Learning), RIFT (the previous state-of-the-art), and various Reinforcement Learning variants like PPO and GRPO.

Key Insights and Ideas

  • The "Alignment" Breakthrough: Previous methods either forced an agent to stay on one path (causing "drift" if the physics didn't match) or let it pick the "most likely" path at every step (causing "mode collapse"). ForSim’s Trajectory-Aligned approach finds the middle ground: it picks the best available path that matches the original intent, ensuring the agent doesn't get "lost" during long simulations.
  • Reactivity is Non-Negotiable: The paper demonstrates that "open-loop" predictions for other agents lead to high collision rates because those agents don't "see" the ego-vehicle coming. By updating predictions stepwise, ForSim creates a "causal link" between observations and actions that is essential for stable RL fine-tuning.
  • Physics-Grounded Consistency: By passing every decision through a PID controller and a kinematic bicycle model, the authors ensure that the resulting trajectories aren't just mathematical lines but represent actual feasible vehicle motion.

Key Results

ForSim-Pluto (the authors' best model) outperformed all baselines across several critical metrics:

  • Safety: It achieved a Collision Per Kilometer (CPK) of 3.95, a massive improvement over the baseline RIFT (6.83) and Pure RL methods (which often exceeded 30.0).
  • Realism: It maintained high scores in kinematic realism (S-SW and S-WD), meaning the simulated driving "looked" like real human data.
  • Comfort: The Uncomfortable Rate (UC)—a measure of jerky or extreme acceleration—dropped to 57.12, significantly lower than RIFT (76.90).
  • Efficiency: The model maintained a high Route Progress (RP) of over 1000, proving it didn't just become "safe" by standing still.

Limitations and Future Work

While ForSim marks a significant step forward, the authors and the methodology suggest a few areas for growth:

  • Computational Overhead: Updating predictions for every agent at every virtual timestep is computationally expensive. Scaling this to hundreds of agents in a dense urban environment might require more optimized "world models."
  • Sim-to-Real Gap: As with all CARLA-based research, the findings are limited by the simulator's internal physics and sensor models. Testing these policies in real-world hardware remains the ultimate hurdle.
  • Predictor Accuracy: The "Stepwise Prediction Rollout" is only as good as the underlying agent prediction model. If the predictor is biased, the "reactive" feedback given to the policy during fine-tuning will also be biased.

Summary updated

Abstract

As the foundation of closed-loop training and evaluation in autonomous driving, traffic simulation still faces two fundamental challenges: covariate shift introduced by open-loop imitation learning and limited capacity to reflect the multimodal behaviors observed in real-world traffic. Although recent frameworks such as RIFT have partially addressed these issues through group-relative optimization, their forward simulation procedures remain largely non-reactive, leading to unrealistic agent interactions within the virtual domain and ultimately limiting simulation fidelity. To address these issues, we propose ForSim, a stepwise closed-loop forward simulation paradigm. At each virtual timestep, the traffic agent propagates the virtual candidate trajectory that best spatiotemporally matches the reference trajectory through physically grounded motion dynamics, thereby preserving multimodal behavioral diversity while ensuring intra-modality consistency. Other agents are updated with stepwise predictions, yielding coherent and interaction-aware evolution. When incorporated into the RIFT traffic simulation framework, ForSim operates in conjunction with group-relative optimization to fine-tune traffic policy. Extensive experiments confirm that this integration consistently improves safety while maintaining efficiency, realism, and comfort. These results underscore the importance of modeling closed-loop multimodal interactions within forward simulation and enhance the fidelity and reliability of traffic simulation for autonomous driving. Project Page: https://currychen77.github.io/ForSim/
BibTeX
@misc{chen2026forsim,
  author = {Keyu Chen and Wenchao Sun and Hao Cheng and Zheng Fu and Sifa Zheng},
  title = {{ForSim: Stepwise Forward Simulation for Traffic Policy Fine-Tuning}},
  year = {2026},
  eprint = {2602.01916},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01916},
}
Figure