While modern Large Language Models (LLMs) and Vision-Language Models (VLMs) can write complex code or describe intricate scenes, they often fail at tasks a human toddler finds trivial. They might correctly identify a glass vase but fail to predict it will shatter if dropped, or they might struggle to determine if a stack of blocks is stable. This discrepancy highlights a fundamental gap in artificial intelligence: the lack of intuitive physics, or the "physical commonsense" required to understand how the world moves, interacts, and persists.
The Developmental Start-up Software
Human intelligence does not begin as a blank slate. As Brenden M. Lake and Joshua B. Tenenbaum argue in Building Machines That Learn and Think Like People, humans possess "developmental start-up software"—innate cognitive building blocks for understanding objects and agency.
Infants, for instance, understand object permanence (objects continue to exist when hidden) and solidity (objects cannot pass through each other). AI models, by contrast, often treat video or images as a sequence of pixel patterns rather than a collection of persistent physical entities. This leads to what Jose Hernandez-Orallo and Anthony G. Cohn 0001 describe as a "mirage" of common sense: models may perform well on linguistic benchmarks while failing elementary spatial reasoning, such as failing to infer that an object inside a moving container moves with that container.
Benchmarking the Consistency Gap
To quantify this deficiency, researchers have developed benchmarks that move beyond simple object recognition.
- PIQA, introduced by Yonatan Bisk and Rowan Zellers, tests "physical commonsense" in natural language, revealing that models struggle to reason about the practical affordances of everyday objects.
- PhysBench, by Wei Chow and colleagues, shows that even advanced models like GPT-4o achieve less than half the accuracy of humans in predicting object dynamics.
- NEWTON, a framework by Yi Ru Wang and Jiafei Duan, exposes a "consistency gap": models might solve complex hypothetical scenarios but fail to identify foundational attributes like whether an object is brittle or heavy.
The Trap of Statistical Shortcuts
Why does scaling up models not solve the problem? Evidence suggests that models are learning statistical shortcuts rather than robust physical laws. Benno Krojer and his team demonstrated this using the Minimal Video Pairs (MVP) benchmark. By presenting models with two visually similar videos that have opposing physical outcomes, they found that AI performance plummeted to near-chance levels (40%), while humans remained highly accurate (92.9%).
Similarly, Luca M. Schulze Buschoff and Konstantinos Voudouris found that fine-tuning VLMs on physical tasks provides only brittle improvements. The models fail to generalize to new visual styles or related cognitive domains, suggesting they are memorizing task-specific patterns rather than internalizing universal principles like gravity or collisions.
Inferring Latent Properties
A key aspect of human physical intuition is online inference—the ability to look at a scene and immediately estimate hidden properties like mass, friction, or elasticity. In Physion++, Hsiao-Yu Tung and Chuang Gan show that current models fail to use early visual observations to predict future outcomes. While a human can hear a sound and know if a vase is ceramic or plastic (Samuel Yu et al., PACS), AI models struggle to bridge the gap between material identification and behavioral prediction.
Toward Object-Centric World Models
To overcome these hurdles, researchers are looking toward architectures that treat the world as a collection of discrete objects rather than raw pixels.
- PLATO, a model developed by Luis S. Piloto, mimics infant learning by representing the world through object-centric experience, successfully acquiring concepts like solidity and continuity.
- Neuro-symbolic frameworks like DUA, proposed by Ludovico Mitchener and Matthew Crosby, combine deep learning with symbolic logic to achieve high efficiency in physical reasoning tasks, requiring far fewer examples than traditional reinforcement learning.
As Jiajun Wu suggests in Learning to See the Physical World, the path forward involves integrating bottom-up perception with top-down physics simulations. Only by moving beyond surface-level pattern recognition can AI achieve the robust, causal understanding of the physical environment necessary for safe and effective real-world interaction.