Driving World Models Move the Camera, Not the Scene

Yuqiao Feng, Xiaoyu Ao

TL;DR

Driving world models often fail to update the 3D scene when given a new plan, effectively moving a virtual camera through a "frozen" world. This paper introduces scene response to quantify this failure and presents ReTime-Compose, a training-free approach that uses geometric transport to force the forecast scene to realistically obey driving commands.

What Problem Does This Paper Address?

Autonomous driving researchers increasingly use "world models" to simulate "what-if" scenarios: What if I brake now? What if I swerve right? For these models to be useful for planning, the predicted video must change realistically. If you slow down, a car ahead should stay farther away for longer than if you maintained speed.

However, the authors discover a fundamental flaw: released world models (like Epona or Orbis 2) move the camera but barely move the scene. They essentially treat the world as a static backdrop that moves with the car. This failure is masked by standard evaluation metrics like scale-aligned Average Displacement Error (ADE), which rescales predicted trajectories to match the command, effectively ignoring whether the model actually obeyed the speed instruction.

Figure 1. Released driving world models move the camera, not the scene, and the trajectory score cannot tell. A: the stereo right camera records the real outcome of a plan 0.54 m to the right (cyan: the tree under the logged path; yellow: where it must move). Released Epona moves it the wrong way; ReTime-Compose on the same frozen Epona follows (unselected KITTI context; all sets in Table 4). B: at half speed the real scene recedes (depth change against the logged speed, from the retimed log); released Orbis 2’s barely changes, ReTime-Compose’s recedes (selected for visibility; all contexts in Sec. 5). C: two real videos under a half-speed command, one that obeys it and one that ignores it (arrows: distance driven, schematic). The scale-aligned ADE of current benchmarks rescales each estimated path before comparing, which removes the speed difference, and favours the video that ignores the command; the scene response α (Sec. 3.1) does not (92 contexts, Fig. 5).

Methods

The authors introduce a new metric called scene response ($\alpha$). It measures the share of real-world depth change that a model's forecast actually reproduces. To get "ground truth" for plans that weren't actually driven, they use two clever references:

  1. Retiming logs: Re-sampling frames from a real drive to simulate different speeds.
  2. Stereo rigs: Using the right-hand camera of a stereo pair as the ground truth for a plan that shifts the car 0.54m to the right.

To fix the "frozen scene" problem, they propose ReTime-Compose, a training-free pipeline:

  • Single Rollout: The model generates one future video at the original "logged" speed.
  • Geometric Transport: A depth estimator and pose-estimation chain are used to "splat" pixels from this rollout into the camera views required by the new plan.
  • Selective Completion: The world model is then used to fill in only the "holes" (disocclusions) created by the camera shift. Crucially, the transported pixels are "pinned" in the model's latent space so the generator cannot overwrite the correct geometry.
The ReTime-Compose pipeline: (a) Roll out once at logged speed. (b) Transport frames to planned cameras using estimated depth. (c) Fill in unseen areas with the frozen world model.

Datasets

The study evaluates five released world models and ten fine-tuned variants across three major autonomous driving datasets:

  • KITTI: The primary dataset for analyzing speed and lateral plans.
  • nuScenes & Argoverse 2: Used to verify that the findings generalize across different sensor setups and environments.

Key Insights and Ideas

  • The Plan-Geometry Mismatch: Models fail because the forecast geometry is dominated by the "context frames" (the past). Plan tokens (the command embeddings) are too weak to override the 3D structure established by the history.
  • ADE is Blind to Speed: Current benchmarks use scale-aligned ADE, which rescales the predicted path. This process removes the speed difference, making it impossible to tell if a model actually slowed down or just ignored the command.
  • Geometry over Tokens: Instead of trying to "teach" a model to move the scene via better training, it is more effective to deliver the plan as explicit geometry through frame warping.
The scale-aligned ADE score (solid line) fails to drop even when a video ignores a command, whereas the proposed scene response (left) correctly tracks execution.

Key Results

  • Baseline Failure: Released models achieve a scene response ($\alpha$) of at most ~0.23, compared to ~0.72 for real video. This means they capture less than a third of the required scene motion.
  • ReTime-Compose Success: For lateral plans, the method reaches 0.93–1.04 of the real stereo response (nearly perfect). For speed plans, it consistently raises the response by +0.11 to +0.24 across different models.
  • Zero-Shot Capability: The method works with "frozen" models—no training is required—and generalizes across KITTI, nuScenes, and Argoverse 2.

Limitations and Future Work

  • Maneuver Scope: The current method focuses on longitudinal (speed) and lateral (shifts) changes. It does not yet handle complex maneuvers like sharp turns or lane changes.
  • Dynamic Agents: While the method handles the ego-vehicle's plan, other agents in the scene (like other cars or pedestrians) are currently kept at their original "physical" time. Future work is needed to realistically re-simulate their reactions to the ego-vehicle's new plan.
  • Closed-Loop Testing: The authors have shown these forecasts are geometrically more accurate, but testing them in a full closed-loop planning system remains a critical future step.

Summary updated

Abstract

Plan-conditioned driving world models are used to compare candidate plans, so their forecast should change with the plan as the real scene would, yet widely used control metrics check only whether the camera follows the command. Recorded drives already contain real outcomes of plans that were never driven: retiming a log gives pose-matched images and LiDAR for other speeds, and a stereo rig records a laterally shifted view. Against these references, released world models move their camera but barely move the scene: their scene response, the share of the real change in distance that the forecast reproduces, reaches at most about a third of real video's, and the trajectory score cannot tell an executed speed command from an ignored one. Interventions inside the generator trace the failure to the plan's channel. The forecast geometry follows the context frames; once they carry the planned geometry, plan tokens add little. ReTime-Compose therefore delivers the plan as geometry, without training. It transports the frames of one logged-speed rollout to the planned cameras, and the frozen model completes only what no frame covers. Frozen generators then follow a 0.54 m lateral shift nearly as the real scene does on KITTI, nuScenes and Argoverse 2, and respond more than re-sampling to slow-down plans in three generators and to speed-up plans in two.
BibTeX
@misc{feng2026driving,
  author = {Yuqiao Feng and Xiaoyu Ao},
  title = {{Driving World Models Move the Camera, Not the Scene}},
  year = {2026},
  eprint = {2611.15260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15260},
}
Figure