Simulating the Visual World with Artificial Intelligence: A Roadmap

Jingtong Yue, Ziqi Huang, Zhaoxi Chen, Xintao Wang, Pengfei Wan, Ziwei Liu

Abstract Recent advancements in video generation, exemplified by models like Sora, have demonstrated remarkable visual fidelity. However, these models often lack a deep understanding of physical consistency and causal reasoning. This paper presents a strategic roadmap for evolving video generation into comprehensive "world models"—simulators capable of modeling physical laws, agent interactions, and complex planning. We introduce a four-generation taxonomy to categorize model capabilities and define a system architecture that separates the implicit world model from the video renderer. This framework serves as a guide for future research in robotics, autonomous driving, and interactive AI.

Introduction: From Pixels to Physics

While modern video generation models have achieved stunning realism, they largely operate as "black-box" pixel generators. They frequently struggle with 3D consistency, long-term cause-and-effect, and adherence to basic physical laws. To bridge this gap, this paper formalizes the concept of a "world model" as a system composed of an implicit world model (the simulation engine) and a video renderer (the visual interface). The goal is to move beyond mere video synthesis toward creating "digital cousins"—generative environments that share the same physical affordances as the real world.

A Four-Generation Taxonomy for World Models

The authors propose a structured evolution for the field, categorized into four distinct generations (G1 through G4). This taxonomy tracks the progression from superficial visual quality to sophisticated, stochastic simulations of reality.

The Four-Generation Taxonomy of World Models, progressing from basic faithfulness to complex planning and stochasticity.
  • G1 - Faithfulness: Focuses on short-term visual quality (2-5 seconds) and superficial motion.
  • G2 - Interactiveness: Introduces controllability through navigation modes and basic 3D reasoning.
  • G3 - Planning: Enables real-time, multi-step task simulation with intrinsic physical plausibility.
  • G4 - Stochasticity: Models rare, low-probability events and operates across vast spatiotemporal scales, from milliseconds to decades.

System Architecture and Navigation Modes

The world modeling process is conceptualized through the mapping of multimodal inputs (text, images, audio, and navigation signals) to video sequences. A critical technical distinction is made between Spatial Conditions and Navigation Modes. While spatial conditions (like depth maps) are tied to specific pixels, Navigation Modes (such as trajectories or action commands) are content-independent. This requires the model to perform internal spatial reasoning to evolve the scene dynamically.

System Overview: The architecture integrates multimodal inputs and navigation conditions into a world model and renderer to produce video and downstream task outputs.

Key Insights: Physical vs. Mental Models

A central theme of the roadmap is the distinction between the Physical World Model and the Mental World Model. The Physical World Model acts as an objective simulator of external dynamics, while the Mental World Model represents an agent's internal reasoning and intentions. The authors argue that establishing a robust physical simulator is the necessary first step toward achieving true artificial intelligence in the physical world.

Furthermore, the paper highlights 2024 as a major inflection point for the field. The transition from U-Net-based diffusion models to Diffusion Transformers (DiT) has provided the scalability required for these more complex simulations. Currently, most state-of-the-art models reside at the G2 level, excelling in interactivity but still lacking the robust planning capabilities required for G3.

Future Challenges in World Simulation

Despite rapid progress, several "grand challenges" remain:

  • Spatiotemporal Scaling: Models must eventually handle both microscopic physical details and macroscopic environmental changes over years.
  • Physical Grounding: Moving from outputs that merely "look right" to those that are mathematically calculated based on physics (e.g., fluid dynamics).
  • Evaluation Metrics: There is a pressing need for benchmarks that evaluate the correctness of a model's internal logic and causal reasoning rather than just its visual output.

Summary updated

Abstract

The landscape of video generation is shifting, from a focus on generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility. These developments point toward the emergence of video foundation models that function not only as visual generators but also as implicit world models, models that simulate the physical dynamics, agent-environment interactions, and task planning that govern real or imagined worlds. This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components: an implicit world model and a video renderer. The world model encodes structured knowledge about the world, including physical laws, interaction dynamics, and agent behavior. It serves as a latent simulation engine that enables coherent visual reasoning, long-term temporal consistency, and goal-driven planning. The video renderer transforms this latent simulation into realistic visual observations, effectively producing videos as a "window" into the simulated world. We trace the progression of video generation through four generations, in which the core capabilities advance step by step, ultimately culminating in a world model, built upon a video generation model, that embodies intrinsic physical plausibility, real-time multimodal interaction, and planning capabilities spanning multiple spatiotemporal scales. For each generation, we define its core characteristics, highlight representative works, and examine their application domains such as robotics, autonomous driving, and interactive gaming. Finally, we discuss open challenges and design principles for next-generation world models, including the role of agent intelligence in shaping and evaluating these systems. An up-to-date list of related works is maintained at this link.
BibTeX
@misc{yue2025simulating,
  author = {Jingtong Yue and Ziqi Huang and Zhaoxi Chen and Xintao Wang and Pengfei Wan and Ziwei Liu},
  title = {{Simulating the Visual World with Artificial Intelligence: A Roadmap}},
  year = {2025},
  eprint = {2511.08585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08585},
}
Figure