Abstract Recent advancements in video generation, exemplified by models like Sora, have demonstrated remarkable visual fidelity. However, these models often lack a deep understanding of physical consistency and causal reasoning. This paper presents a strategic roadmap for evolving video generation into comprehensive "world models"—simulators capable of modeling physical laws, agent interactions, and complex planning. We introduce a four-generation taxonomy to categorize model capabilities and define a system architecture that separates the implicit world model from the video renderer. This framework serves as a guide for future research in robotics, autonomous driving, and interactive AI.
Introduction: From Pixels to Physics
While modern video generation models have achieved stunning realism, they largely operate as "black-box" pixel generators. They frequently struggle with 3D consistency, long-term cause-and-effect, and adherence to basic physical laws. To bridge this gap, this paper formalizes the concept of a "world model" as a system composed of an implicit world model (the simulation engine) and a video renderer (the visual interface). The goal is to move beyond mere video synthesis toward creating "digital cousins"—generative environments that share the same physical affordances as the real world.
A Four-Generation Taxonomy for World Models
The authors propose a structured evolution for the field, categorized into four distinct generations (G1 through G4). This taxonomy tracks the progression from superficial visual quality to sophisticated, stochastic simulations of reality.
- G1 - Faithfulness: Focuses on short-term visual quality (2-5 seconds) and superficial motion.
- G2 - Interactiveness: Introduces controllability through navigation modes and basic 3D reasoning.
- G3 - Planning: Enables real-time, multi-step task simulation with intrinsic physical plausibility.
- G4 - Stochasticity: Models rare, low-probability events and operates across vast spatiotemporal scales, from milliseconds to decades.
System Architecture and Navigation Modes
The world modeling process is conceptualized through the mapping of multimodal inputs (text, images, audio, and navigation signals) to video sequences. A critical technical distinction is made between Spatial Conditions and Navigation Modes. While spatial conditions (like depth maps) are tied to specific pixels, Navigation Modes (such as trajectories or action commands) are content-independent. This requires the model to perform internal spatial reasoning to evolve the scene dynamically.
Key Insights: Physical vs. Mental Models
A central theme of the roadmap is the distinction between the Physical World Model and the Mental World Model. The Physical World Model acts as an objective simulator of external dynamics, while the Mental World Model represents an agent's internal reasoning and intentions. The authors argue that establishing a robust physical simulator is the necessary first step toward achieving true artificial intelligence in the physical world.
Furthermore, the paper highlights 2024 as a major inflection point for the field. The transition from U-Net-based diffusion models to Diffusion Transformers (DiT) has provided the scalability required for these more complex simulations. Currently, most state-of-the-art models reside at the G2 level, excelling in interactivity but still lacking the robust planning capabilities required for G3.
Future Challenges in World Simulation
Despite rapid progress, several "grand challenges" remain:
- Spatiotemporal Scaling: Models must eventually handle both microscopic physical details and macroscopic environmental changes over years.
- Physical Grounding: Moving from outputs that merely "look right" to those that are mathematically calculated based on physics (e.g., fluid dynamics).
- Evaluation Metrics: There is a pressing need for benchmarks that evaluate the correctness of a model's internal logic and causal reasoning rather than just its visual output.
Summary updated