Keeping Reactive Traffic Where the Reconstruction Can See

Driving simulators built from real sensor logs now look photorealistic. Methods such as Gaussian splatting build a 3D model of a recorded street out of many small colored blobs that can be re-rendered from new camera positions. The catch is that the other cars and pedestrians in these simulators are puppets. They replay their recorded paths no matter what the tested car does. Traffic models that do react to the tested car exist, but they only output coordinates on a map and produce no images. The obvious fix is to let a learned traffic model drive the reconstructed actors. That fix hides a problem. Every reconstructed actor, and the patch of background behind it, was only ever seen from the poses it actually occupied in the log. Once a reactive policy moves a car somewhere new, for example yielding into a lane it never used or braking early enough to reveal road it never uncovered, the renderer has to invent pixels. A camera-based planner then reacts to rendering artifacts instead of to traffic.

This note proposes a simulator in which each actor extracted from a log becomes two things at once: a renderable 4D asset and an agent with a learned reactive policy. The policy is constrained by a renderability envelope, a per-actor, per-time estimate of how badly the scene will render if that actor is displaced to a given pose. Reactive traffic then deviates from the log only as far as the reconstruction can support. Every simulated episode reports how much of its interaction happened inside trusted territory. The scientific question is whether reactivity changes how we judge end-to-end driving policies, and whether it can do so without rendering error confounding the answer.

Core Claim

Log-derived closed-loop simulation has a hidden trade-off between how much actors react and how faithfully the scene renders. That trade-off can be measured from the log itself and turned into an explicit constraint on the traffic policy. We claim three things. First, rendering error at displaced actor poses can be predicted from observation coverage alone. Coverage means the viewing angles and distances from which the actor and its occluded background were seen. Second, a reactive policy trained with this predictor as a soft penalty keeps most of the behavioral realism of an unconstrained policy while cutting rendering error at the poses the planner actually sees. Third, when planners are tested against constrained reactive traffic, their failure rates and rankings differ measurably from both scripted-hazard testing and unconstrained reactive testing. The differences with the unconstrained version show how much of an apparent planner failure was caused by artifacts.

Relation to Prior Work

The rendering side is supplied by static-dynamic decompositions that split a street into a fixed background and separate moving objects. UniSim builds a background voxel grid plus per-actor feature fields and lets the ego vehicle deviate from its logged path. Street Gaussians does the same with background and per-object Gaussians and renders at 135 FPS. Street Gaussians without 3D Object Tracker removes the need for ground-truth 3D boxes by lifting 2D foundation-model tracks and learning corrective per-point motion. We reuse the last pipeline for asset extraction. In all three systems, actors are replayed or hand-scripted.

The motivation comes from closed-loop testing. NeuroNCAP edits hazards into neural reconstructions of nuScenes and finds 88% to 92% failure rates for raw UniAD and VAD outputs, but its hazards follow scripted open-loop paths. Pseudo-Simulation pre-renders splatted views of perturbed ego poses and reaches R^2 = 0.8 correlation with closed-loop scores. It explicitly cannot handle actors that respond.

The behavior side is supplied by ProSim, a closed-loop multi-agent policy trained on the Waymo Open Motion Dataset. It reports 0.28 m ADE (average displacement error) under full prompting and runs 64 agents in 50 ms, but it operates purely in map coordinates. LookOut shows that joint rather than independent futures matter for planning, and in doing so constrains our design: the policy must be joint, not per-agent car-following.

GeoSim is the one prior system that inserts reactive actors into real footage, but its actors follow the Intelligent Driver Model (IDM), a one-dimensional car-following rule. DriveArena couples a traffic manager with a diffusion video generator. It abandons explicit geometry, and UniAD reaches only 13.7% route completion there. None of these systems asks where the renderer can be trusted when actors react. That question is the contribution here.

Research Program

We first build the asset-agent pairs. For each log in Waymo-NOTR dynamic32 and in the 14 nuScenes sequences used by NeuroNCAP, we run the tracker-free Gaussian pipeline. This yields a background field and per-actor Gaussian sets with canonical-frame poses over time. For each actor we also record its coverage: the set of relative camera-to-actor viewing directions and distances observed. For each background region, we record the times at which it was visible rather than occluded.

Next we learn the renderability predictor. Within each log we hold out frames and camera views, render actors at poses that were observed but withheld from training, and record photometric error on the actor mask and on newly revealed background. A small regressor maps coverage features to expected error: angular and distance gap to the nearest observed view, fraction of newly revealed background never observed, and actor class. This gives a self-supervised, per-pose error estimate $\hat{e}_i(p, t)$ for actor $i$ at pose $p$ and time $t$. No ground truth for unseen poses is needed. The assumption that held-out error extrapolates to truly novel poses is tested directly (see below).

Third, we fine-tune a ProSim-style joint policy on WOMD with an added penalty:

$$\mathcal{L} = \mathcal{L}_{\text{imitation}} + \lambda \sum_{i,t} \max\big(0,\ \hat{e}_i(p_{i,t}, t) - \tau\big),$$

where $\mathcal{L}_{\text{imitation}}$ is the original closed-loop imitation loss, $p_{i,t}$ is the pose the policy assigns actor $i$ at step $t$, $\tau$ is an acceptable error level, and $\lambda$ trades reactivity against fidelity. Because WOMD logs lack imagery, $\hat{e}$ during training is computed from geometric coverage proxies. On the reconstructed logs the full predictor is used at test time as a guidance term. Sweeping $\lambda$ traces the reactivity-fidelity curve that we want to characterize.

Finally, we wrap the renderer, the policy, and a kinematic ego model into a closed loop running at 10 Hz and plug in UniAD and VAD. The simulator also logs, per episode, the fraction of planner-visible frames in which some actor exceeded $\tau$.

Baselines

The naive reference point is log replay with a renderable ego. Other actors follow their recorded paths, as in UniSim or Street Gaussians. It is run on the same reconstructions, so only the behavior layer differs. It anchors how often planners fail when traffic ignores them.

The scripted-hazard reference is NeuroNCAP's protocol on its 14 nuScenes sequences, which reported 88% to 92% raw-output failure rates for UniAD and VAD. We rerun it in our renderer so that reactive variants of the same hazards can be compared one-to-one.

The rule-based reactive baseline is IDM attached to reconstructed actors, the GeoSim approach. It is run with the same assets and the same ego models. DeepMF reports an 88% collision rate for IDM planners under adversarial cut-ins, which suggests IDM traffic is too naive to be a credible stand-in for human reactions.

The strongest competitor is the unconstrained ProSim-style policy driving the same Gaussian actors, which is the $\lambda = 0$ ablation. It is the obvious engineering combination and isolates our key ingredient. It uses the same WOMD training data, the same number of fine-tuning steps and hyperparameter search budget, and the same rendering. Its behavioral reference number is ProSim's 0.28 m ADE on ProSim-Instruct-520k. For rendering at novel viewpoints, the reference is Street Gaussians without Tracker's reported gain of 2.66 dB DPSNR over S3Gaussian on Waymo-NOTR. We would count the proposal as beating this baseline under two conditions. First, at matched behavioral realism (no more than a 10% relative increase in WOMD joint minADE, the minimum average displacement error over sampled futures), the constrained policy must reduce predicted and measured actor-region photometric error on planner-visible frames. Second, planner decisions must flip less often when artifacts are removed (see the decisive checks).

The generative baseline is DriveArena. We compare against its reported closed-loop numbers only, 13.7% route completion and an Arena Driving Score of 0.086 for UniAD, because retraining it is outside our compute budget.

Why not just…?

Why not just bolt ProSim onto Street Gaussians and skip the envelope? This is the real threat. If camera planners turn out to be insensitive to the artifacts produced by displaced actors, the envelope adds cost and no benefit. The $\lambda = 0$ versus $\lambda > 0$ comparison settles it, together with the artifact-sensitivity check below. If planner outputs barely change when displaced actors are rendered badly, we will report that the simple combination suffices and that the envelope is only a diagnostic.

Why not just use IDM on top of the reconstruction? IDM only models following along lane centerlines, so it cannot produce the yields, nudges, and negotiated merges that the interaction literature identifies as the source of planning failures (LookOut). IDM is included as a baseline. If its planner failure rates match the learned policy's, the learned behavior layer is unnecessary for these benchmarks.

Why not just use a video diffusion world model? ReSim needs about two minutes on an A100 per 4-second clip, which rules out 10 Hz closed loops. DriveArena reports temporal drift and very low closed-loop completion. Diffusion models also provide no per-actor handle on where the output can be trusted, which is exactly the quantity we want to measure.

Why not just evaluate in vector space, as ProSim and LookOut do? Vector-space evaluation cannot test end-to-end camera planners. The copycat failures NeuroNCAP documents arise between perception and planning, so reproducing them requires pixels.

Decisive Checks

The first check tests whether the envelope's predictions hold for genuinely novel poses. On Waymo segments where the same street is traversed by several actors, one actor's asset is rendered at poses another actor occupied and compared against the real frames. The check passes if the rank correlation between $\hat{e}$ and measured error is high. Failure here would invalidate the envelope.

The second check measures artifact sensitivity. For each closed-loop episode we re-render the planner's inputs with displaced actors replaced by their nearest in-coverage appearance and record how often the planner's first-second action changes. If the unconstrained policy produces many more such flips than the constrained one, rendering error was confounding planner evaluation.

The third check tests whether reactivity changes conclusions. On NeuroNCAP-style hazards and on interaction scenarios mined from nuScenes logs, we compare UniAD and VAD failure rates and their relative ranking under replay, IDM, unconstrained reactive traffic, and constrained reactive traffic. We also test whether pseudo-simulation scores computed from the same scenes correlate with the constrained closed-loop scores as well as the reported R^2 = 0.8 does for rule-based traffic. A ranking change that survives the artifact-sensitivity control would show that scripted-hazard testing mismeasures these planners.