When testing autonomous systems in virtual environments, the utility of the simulation hinges entirely on how realistically the surrounding agents behave. For years, simulators relied on hand-crafted, rule-based heuristics to govern background traffic, crowds, or opponents. While these models provide high stability and clear interpretability, they suffer from a fundamental limitation: they lack the mathematical expressivity required to capture the diverse, subtle, and often messy nuances of real-world human behavior (Arbabi et al., 2022).
This discrepancy creates a behavioral realism gap. When virtual agents behave like predictable robots, the autonomous systems being tested learn to exploit these rigid rules rather than preparing for the complex social interactions they will face in reality. To bridge this gap, a major research direction has emerged to replace hand-crafted heuristics with data-driven, generative behavior models.
Why Rule-Based Heuristics Fail
Traditional traffic and crowd simulators use deterministic rules—such as simple gap-keeping, lane-following logic, or basic pathfinding algorithms—to move background agents. However, as Southey et al. (2007) observe, simple pathfinding models struggle to account for non-optimal human decision-making and real-world environmental constraints.
In dense, interactive settings like a busy metropolitan intersection, human behavior is not governed by rigid equations; it is a process of informal negotiation, social norms, and multi-second strategic planning (Zang and Turkcan, 2024). Hand-crafted procedural rules fail to capture common but irregular actions like double-parking, lane-weaving, or jaywalking, resulting in overly sanitized environments (Tan et al., 2021; Ludlow et al., 2024).
[Traditional Rule-Based Simulation]
Situation: Vehicle approaches a merging car.
Action: Apply deterministic deceleration formula -> Predictable, robotic response.
[Data-Driven Generative Simulation]
Situation: Vehicle approaches a merging car.
Action: Sample from learned human trajectory distribution -> Multi-modal responses (yield, accelerate, or nudge).
When background traffic follows simplified logic, it fails to replicate the aggressive or unpredictable maneuvers encountered on public roads (Yan et al., 2025; Karnchanachari et al., 2024). As Althoff (2010) noted, transitioning from these expert-defined heuristics to data-driven learning approaches is necessary to capture the true stochastic nature of real-world environments. This limitation extends beyond driving; conventional sports simulators and virtual human-assistant platforms similarly suffer from a performance gap when their rule-based agents fail to mirror tactical nuances or complex social interactions (Wang et al., 2024; Xuan et al., 2025).
Capturing Multi-Modality and Realism from Data
The core insight of modern behavior modeling is that instead of trying to manually program every potential human reaction, simulators should learn a probability distribution over actions directly from real-world demonstrations.
To achieve this, researchers deploy generative architectures that can capture the multi-modal nature of human behavior—the fact that given the exact same traffic scenario, one driver might aggressively accelerate while another politely yields. For instance, TrafficGen, developed by Feng et al. (2022), pairs an autoregressive vehicle placement strategy with a multi-modal trajectory generator to synthesize diverse traffic layouts from real-world driving logs. Similarly, Jiang et al. (2026) introduce a probabilistic planning paradigm (VADv2) that discretizes the action space into a "planning vocabulary," predicting a probability distribution over trajectories to handle the inherent uncertainty of human driving.
This data-driven paradigm allows simulators to scale up testing by generating realistic variations of scenarios. Even when high-resolution data is unavailable, frameworks like ImIn-GAIL integrate trajectory interpolation directly into imitation learning to extract realistic behaviors from sparse, low-resolution urban data sources (Wei et al., 2021). The approach is flexible enough to capture the erratic, subtle behavioral nuances of biological swarms directly from raw video footage, bypassing the need for explicit 3D motion tracking (Wu et al., 2025).
The Challenge of Closed-Loop Interaction
Populating a scene with realistic initial positions or replaying historical logs (open-loop simulation) is insufficient for long-term testing. If the autonomous vehicle under test deviates from the historical script, the background agents must react dynamically and plausibly (closed-loop simulation). As Chitta et al. (2024) point out in their work on SLEDGE, a persistent gap remains between static realism (synthesizing a realistic initial snapshot) and dynamic behavioral realism (how those agents interact over time).
To ensure that generative models remain reactive and socially consistent, researchers incorporate multi-agent coordination mechanisms and specialized loss functions. TrafficSim, proposed by Suo et al. (2021), leverages a fully differentiable simulation framework combined with a "common sense" collision loss. This allows the model to learn complex, socially-consistent behaviors like merging and yielding directly from human demonstrations, avoiding the robotic, rigid reactions of heuristic models (Gao et al., 2022).
[Open-Loop Replay]
Ego Vehicle swerves -> Background Agent blindly follows historical log -> Unrealistic Collision
[Closed-Loop Reactive Model]
Ego Vehicle swerves -> Background Agent senses deviation -> Generative model updates trajectory -> Realistic Evasion
When modeling dense collective systems, agents constantly shift between cooperative and competitive behaviors. To capture this fluid dynamic, Coordinated Policy Optimization (CoPO) utilizes multi-agent reinforcement learning with social psychology principles to balance individual and neighborhood rewards, generating highly realistic collective flows (Peng et al., 2021).
Balancing Control, Structure, and Safety
While pure data-driven models excel at diversity, they can suffer from compounding errors, occasionally generating physically impossible trajectories or ignoring traffic regulations (Finkeldei and Thees, 2025). Furthermore, engineers need to be able to control the simulation to test specific edge cases. The field has addressed this by developing hybrid frameworks that overlay control mechanisms onto generative models.
One prominent approach is the use of diffusion-based planners with guidance functions. LCSim integrates a diffusion motion planner with customizable guide functions, allowing users to steer the simulation from gentle city driving to adversarial edge cases across city-scale environments (Ding et al., 2024). For safety-critical testing, DiffScene steers the diffusion denoising process with adversarial gradients to deliberately synthesize rare, high-risk events that remain physically plausible (Xu et al., 2025).
Another approach is character editing, which breaks the assumption that all human drivers behave identically. The SCBG framework introduces a differentiable courtesy metric that allows users to explicitly "edit" the personality of simulated drivers, generating a spectrum of behaviors from highly cooperative to entirely selfish (Chang et al., 2023).
To evaluate these advancements, benchmarks like the Waymo Open Sim Agents Challenge (WOSAC) have shifted the industry standard away from raw sensor fidelity toward mid-level object trajectories, utilizing multi-faceted realism metrics to rigorously assess how well AI agents mimic the multi-modal distribution of human interaction (Anguelov and Montali, 2023). By combining the expressive power of generative modeling with structured physical and social constraints, this research direction continues to narrow the sim-to-real gap, creating virtual worlds that match the behavioral complexity of the real one.
Key Foundational Work
Reachability Analysis and its Application to the Safety Assessment of Autonomous Cars (2010) — contributed 1 concept elements to this direction.
Inferring Complex Agent Motions from Partial Trajectory Observations (2007) — contributed 1 concept elements to this direction.