TL;DR
Driving world models often fail to update the 3D scene when given a new plan, effectively moving a virtual camera through a "frozen" world. This paper introduces scene response to quantify this failure and presents ReTime-Compose, a training-free approach that uses geometric transport to force the forecast scene to realistically obey driving commands.
What Problem Does This Paper Address?
Autonomous driving researchers increasingly use "world models" to simulate "what-if" scenarios: What if I brake now? What if I swerve right? For these models to be useful for planning, the predicted video must change realistically. If you slow down, a car ahead should stay farther away for longer than if you maintained speed.
However, the authors discover a fundamental flaw: released world models (like Epona or Orbis 2) move the camera but barely move the scene. They essentially treat the world as a static backdrop that moves with the car. This failure is masked by standard evaluation metrics like scale-aligned Average Displacement Error (ADE), which rescales predicted trajectories to match the command, effectively ignoring whether the model actually obeyed the speed instruction.
Methods
The authors introduce a new metric called scene response ($\alpha$). It measures the share of real-world depth change that a model's forecast actually reproduces. To get "ground truth" for plans that weren't actually driven, they use two clever references:
- Retiming logs: Re-sampling frames from a real drive to simulate different speeds.
- Stereo rigs: Using the right-hand camera of a stereo pair as the ground truth for a plan that shifts the car 0.54m to the right.
To fix the "frozen scene" problem, they propose ReTime-Compose, a training-free pipeline:
- Single Rollout: The model generates one future video at the original "logged" speed.
- Geometric Transport: A depth estimator and pose-estimation chain are used to "splat" pixels from this rollout into the camera views required by the new plan.
- Selective Completion: The world model is then used to fill in only the "holes" (disocclusions) created by the camera shift. Crucially, the transported pixels are "pinned" in the model's latent space so the generator cannot overwrite the correct geometry.
Datasets
The study evaluates five released world models and ten fine-tuned variants across three major autonomous driving datasets:
- KITTI: The primary dataset for analyzing speed and lateral plans.
- nuScenes & Argoverse 2: Used to verify that the findings generalize across different sensor setups and environments.
Key Insights and Ideas
- The Plan-Geometry Mismatch: Models fail because the forecast geometry is dominated by the "context frames" (the past). Plan tokens (the command embeddings) are too weak to override the 3D structure established by the history.
- ADE is Blind to Speed: Current benchmarks use scale-aligned ADE, which rescales the predicted path. This process removes the speed difference, making it impossible to tell if a model actually slowed down or just ignored the command.
- Geometry over Tokens: Instead of trying to "teach" a model to move the scene via better training, it is more effective to deliver the plan as explicit geometry through frame warping.
Key Results
- Baseline Failure: Released models achieve a scene response ($\alpha$) of at most ~0.23, compared to ~0.72 for real video. This means they capture less than a third of the required scene motion.
- ReTime-Compose Success: For lateral plans, the method reaches 0.93–1.04 of the real stereo response (nearly perfect). For speed plans, it consistently raises the response by +0.11 to +0.24 across different models.
- Zero-Shot Capability: The method works with "frozen" models—no training is required—and generalizes across KITTI, nuScenes, and Argoverse 2.
Limitations and Future Work
- Maneuver Scope: The current method focuses on longitudinal (speed) and lateral (shifts) changes. It does not yet handle complex maneuvers like sharp turns or lane changes.
- Dynamic Agents: While the method handles the ego-vehicle's plan, other agents in the scene (like other cars or pedestrians) are currently kept at their original "physical" time. Future work is needed to realistically re-simulate their reactions to the ego-vehicle's new plan.
- Closed-Loop Testing: The authors have shown these forecasts are geometrically more accurate, but testing them in a full closed-loop planning system remains a critical future step.
Summary updated