Critiques of World Models

Eric P. Xing, Mingkai Deng, Jinyu Hou, Zhiting HuarXiv 2025 · DOI · arXiv

Abstract

This paper argues that "World Models" should be viewed as internal sandboxes for purposeful reasoning and acting, rather than just high-fidelity video generators. The authors critique current trends—specifically the move away from generative reconstruction and discrete tokens—and propose PAN (Physical, Agentic, and Nested), a new hierarchical architecture that combines the strengths of LLMs with continuous sensory prediction.

Introduction: The Identity Crisis of World Models

In the race toward AGI, the term "World Model" has become a buzzword with conflicting definitions. Some see it as a system that generates realistic video (like OpenAI’s Sora), while others, most notably the "vocal school of thought" led by Yann LeCun, argue for non-generative, latent-only architectures (JEPA) to avoid the "doomed" nature of autoregressive models.

The authors identify a critical gap: current world models either focus too much on visual "eye candy" (which lacks interactivity) or attempt to bypass data reconstruction entirely. The latter can lead to "representation collapse"—where a model "cheats" by mapping all inputs to a single constant value to minimize loss without learning anything about the world.

Technical Critique: The Fallacy of Latent-Only Models

A core contribution of this work is the formal critique of latent reconstruction loss. The authors provide Theorem 2, demonstrating that latent reconstruction loss is merely an upper bound for generative loss. Minimizing latent loss does not guarantee consistency with the real world unless the encoder and decoder are perfect inverses.

Figure 4: Latent reconstruction loss in latent space, which the authors argue is prone to representation collapse without generative grounding.

Without a decoder to "prove" what the model knows by reconstructing an image or sound, the encoder and world model can simply agree to output zeros, achieving zero loss while learning zero information. Generative reconstruction provides a stable anchor that prevents these trivial solutions.

The PAN Architecture: Physical, Agentic, and Nested

To address these issues, the paper proposes the PAN architecture. This model is designed to be a "reasoning sandbox" that handles multiple scales of abstraction simultaneously.

Figure 3: The PAN architecture uses internal simulative reasoning (T') to find best actions based on an agent's belief state, grounded in actual observations (T).

Key features of PAN include:

  • Mixed Representations: Using both discrete tokens for symbolic reasoning and continuous embeddings for physical nuance. Theorem 1 proves that discrete tokens can represent continuous data with arbitrary precision by "scaling out" sequence length.
  • Generative Latent Prediction (GLP): A hierarchical approach where an LLM backbone handles long-term "thought space" while a diffusion-based module handles low-level sensory dynamics.
  • Nested Reasoning: Illustrated by a mountaineering expedition, where a climber needs high-level "thought" (route planning) and low-level "subconscious" reaction (shifting weight on an icepick) at the same time.

Key Insights

  • Text is Compressed Experience: Language is a highly evolved compression of human experience, capturing social norms and counterfactuals that are invisible in raw video.
  • Generative Models aren't "Doomed": While critics argue that autoregressive models accumulate error, the authors note that many real-world systems are inherently chaotic. A good world model should aim for "ergodic" stability—learning abstract properties that remain predictable.
  • RL > MPC for Foresight: Reinforcement Learning (RL) using a world model as a simulator allows for much longer horizons and faster decision-making compared to standard Model Predictive Control (MPC).

Conclusion

The path to AGI lies not in avoiding generation, but in mastering it through hierarchical, grounded simulation. The PAN architecture represents a shift toward world models that are not just visualizers, but active reasoning engines capable of navigating the complexity of the physical and social world.

Summary updated

BibTeX
@misc{xing2025critiques,
  author = {Eric P. Xing and Mingkai Deng and Jinyu Hou and Zhiting Hu},
  title = {{Critiques of World Models}},
  year = {2025},
  eprint = {2507.05169},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05169},
}
Figure