ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos

Xiaodong Wang, Peixi PengarXiv 2025 · DOI · arXiv

Abstract

ProphetDWM is an end-to-end driving world model designed to jointly predict future video frames and driving actions in a single stage. By integrating a lightweight action module with a diffusion-based transition module, it overcomes the error accumulation typical of two-stage models. Evaluated on the NuScenes dataset, ProphetDWM achieves state-of-the-art performance in video quality and temporal consistency, enabling stable long-term "what-if" simulations for autonomous driving.

Introduction

World models are essential for autonomous vehicles to simulate future scenarios based on potential actions. However, current models suffer from three primary limitations: the "Two-Stage Trap," where separate video and action prediction leads to error accumulation; "Action Dependency," requiring manual action inputs for every frame; and a lack of focus on the underlying "dynamical laws" of vehicle movement.

Comparison of ProphetDWM (Ours) with existing two-stage models like ADriver-I and DriveDreamer.

ProphetDWM addresses these by unifying video and action prediction into a single task, ensuring physical grounding and temporal stability.

Methodology

ProphetDWM's architecture consists of two primary components working in tandem to provide a unified rollout of future states.

The ProphetDWM architecture featuring a light-weight action module and a video prediction module for joint rollout.

Light-weight Action Module

The model uses a Multi-Layer Perceptron (MLP) based module to learn latent actions. By processing current visual observations and a short history of past actions, it predicts high-dimensional features representing vehicle intent. This allows the model to autonomously "roll out" future speed and steering signals without requiring per-frame manual inputs.

Diffusion-based Transition Module

For video generation, ProphetDWM employs a spatiotemporal U-Net diffusion model. It utilizes the latent actions as guidance to ensure the generated visual sequence adheres to predicted physics. To prevent "drifting"—where scene elements change inconsistently over time—a State Context Short Pathway injects multi-scale features from the initial frame directly into the U-Net's hidden states, serving as a visual anchor.

Experiments and Results

The model was trained on the NuScenes benchmark. To enhance scene understanding, the authors utilized LLaVA-NeXT-Video to generate descriptive text for training videos, providing a richer semantic signal for the diffusion backbone.

ProphetDWM established new state-of-the-art results on NuScenes:

  • Video Quality: Achieved a Fréchet Inception Distance (FID) of 6.9, significantly outperforming Drive-WM (23.3) and GenAD (15.4).
  • Temporal Consistency: Reached a Fréchet Video Distance (FVD) of 190.5, the best among open-source models.
  • Action Accuracy: Recorded the lowest L1 error in speed prediction (0.090) and competitive steering accuracy (0.093).
  • Long-term Stability: Demonstrated the ability to generate consistent rollouts for up to 100 frames (~25 seconds), simulating complex maneuvers like lane deviations and traffic responses.

Conclusion

ProphetDWM demonstrates that joint learning of pixels and actions leads to more physically grounded world models. While highly effective, future work will focus on scaling to 360-degree multi-view consistency and integrating reward modeling to transition from a predictive model to a full reinforcement learning agent.

Summary updated

Abstract

Real-world driving requires people to observe the current environment, anticipate the future, and make appropriate driving decisions. This requirement is aligned well with the capabilities of world models, which understand the environment and predict the future. However, recent world models in autonomous driving are built explicitly, where they could predict the future by controllable driving video generation. We argue that driving world models should have two additional abilities: action control and action prediction. Following this line, previous methods are limited because they predict the video requires given actions of the same length as the video and ignore the dynamical action laws. To address these issues, we propose ProphetDWM, a novel end-to-end driving world model that jointly predicts future videos and actions. Our world model has an action module to learn latent action from the present to the future period by giving the action sequence and observations. And a diffusion-model-based transition module to learn the state distribution. The model is jointly trained by learning latent actions given finite states and predicting action and video. The joint learning connects the action dynamics and states and enables long-term future prediction. We evaluate our method in video generation and action prediction tasks on the Nuscenes dataset. Compared to the state-of-the-art methods, our method achieves the best video consistency and best action prediction accuracy, while also enabling high-quality long-term video and action generation.
BibTeX
@misc{wang2025prophetdwm,
  author = {Xiaodong Wang and Peixi Peng},
  title = {{ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos}},
  year = {2025},
  eprint = {2505.18650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18650},
}
Figure