Abstract
ProphetDWM is an end-to-end driving world model designed to jointly predict future video frames and driving actions in a single stage. By integrating a lightweight action module with a diffusion-based transition module, it overcomes the error accumulation typical of two-stage models. Evaluated on the NuScenes dataset, ProphetDWM achieves state-of-the-art performance in video quality and temporal consistency, enabling stable long-term "what-if" simulations for autonomous driving.
Introduction
World models are essential for autonomous vehicles to simulate future scenarios based on potential actions. However, current models suffer from three primary limitations: the "Two-Stage Trap," where separate video and action prediction leads to error accumulation; "Action Dependency," requiring manual action inputs for every frame; and a lack of focus on the underlying "dynamical laws" of vehicle movement.
ProphetDWM addresses these by unifying video and action prediction into a single task, ensuring physical grounding and temporal stability.
Methodology
ProphetDWM's architecture consists of two primary components working in tandem to provide a unified rollout of future states.
Light-weight Action Module
The model uses a Multi-Layer Perceptron (MLP) based module to learn latent actions. By processing current visual observations and a short history of past actions, it predicts high-dimensional features representing vehicle intent. This allows the model to autonomously "roll out" future speed and steering signals without requiring per-frame manual inputs.
Diffusion-based Transition Module
For video generation, ProphetDWM employs a spatiotemporal U-Net diffusion model. It utilizes the latent actions as guidance to ensure the generated visual sequence adheres to predicted physics. To prevent "drifting"—where scene elements change inconsistently over time—a State Context Short Pathway injects multi-scale features from the initial frame directly into the U-Net's hidden states, serving as a visual anchor.
Experiments and Results
The model was trained on the NuScenes benchmark. To enhance scene understanding, the authors utilized LLaVA-NeXT-Video to generate descriptive text for training videos, providing a richer semantic signal for the diffusion backbone.
ProphetDWM established new state-of-the-art results on NuScenes:
- Video Quality: Achieved a Fréchet Inception Distance (FID) of 6.9, significantly outperforming Drive-WM (23.3) and GenAD (15.4).
- Temporal Consistency: Reached a Fréchet Video Distance (FVD) of 190.5, the best among open-source models.
- Action Accuracy: Recorded the lowest L1 error in speed prediction (0.090) and competitive steering accuracy (0.093).
- Long-term Stability: Demonstrated the ability to generate consistent rollouts for up to 100 frames (~25 seconds), simulating complex maneuvers like lane deviations and traffic responses.
Conclusion
ProphetDWM demonstrates that joint learning of pixels and actions leads to more physically grounded world models. While highly effective, future work will focus on scaling to 360-degree multi-view consistency and integrating reward modeling to transition from a predictive model to a full reinforcement learning agent.
Summary updated