To understand why text-only language models struggle with the physical world, consider a famous thought experiment revived by Cory Paik et al. in The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color: an octopus underwater intercepts a communication cable between two islands and listens to all human conversations. The octopus learns the statistical patterns of language perfectly, but it has never seen a banana, felt gravity, or experienced light.
When humans talk or write, they rarely mention the completely obvious. This phenomenon is known as reporting bias. As Vered Shwartz et al. demonstrate in Do Neural Language Models Overcome Reporting Bias?, text corpora heavily over-represent rare or sensational events (like "murder") while under-representing trivial, everyday realities (like "breathing").
This creates a skewed distribution for a text-only model. For instance, humans frequently write "green banana" to distinguish an unripened fruit from a normal one, but they rarely write "yellow banana" because yellow is the default assumption. Consequently, text-only models often conclude that bananas are predominantly green. While massive scaling can help mitigate some of these surface-level frequency biases—as observed by Fangyu Liu et al. in Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour—the underlying issue remains: text-only models lack a mechanism to verify linguistic tokens against physical reality.
The Representational Ceiling of Text
If text-only models are just missing data, can we solve the problem by simply scaling them up? The evidence suggests a hard boundary.
Yonatan Bisk et al. introduced a benchmark in PIQA: Reasoning about Physical Commonsense in Natural Language to evaluate physical commonsense, revealing a persistent performance gap between humans and text-only models. This gap isn't just a matter of parameter count. Lei Li et al. found in Can Language Models Understand Physical Concepts? that while scaling up text-only models improves their ability to associate words with visual properties like color or material, it completely plateaus when it comes to non-visual embodied concepts like mass, hardness, or temperature. A model can read a million sentences about "heavy rocks," but without sensorimotor interaction, it cannot ground the concept of mass in actual physical sensation.
Similarly, Ari Holtzman et al. demonstrate in Do Neural Language Representations Learn Physical Commonsense? that text-only models fail to connect an object's intrinsic properties (e.g., rigidity) to its physical affordances (e.g., can be shattered). The models learn the statistical co-occurrence of words but fail to build an internal "world model" that understands natural laws. As Jared Moore argues in Language Models Understand Us, Poorly, text-only training leads to a fundamental under-specification where abstract concepts lack the environmental and physical context required for true comprehension. Furthermore, Bruce W. Lee et al. show in Language Models Don't Learn the Physical Manifestation of Language that scaling fails to teach models even the physical manifestation of language itself—such as how characters look or sound—proving that the limitation is inherent to the text-only paradigm rather than computational capacity.
Grounding Through Imagination and Simulation
To break through this representational ceiling, researchers are exploring ways to inject perceptual grounding back into language models.
One approach is to let the model "imagine" the visual world. For example, the framework proposed by Guy Yariv et al. in Improving Visual Commonsense in Language Models via Multiple Image Generation allows language models to generate multiple synthetic images internally to reason about physical properties like shape and relative size. By fusing these imagined visual features with text representations, the model gains a surrogate form of visual commonsense.
Another powerful paradigm moves beyond static images into dynamic physical environments. Ruibo Liu et al. developed a method in Mind's Eye: Grounded Language Model Reasoning through Simulation that grounds language reasoning by using a physics engine. When a model faces a question about physical interactions, it translates the text into simulation code, runs the simulation in a virtual environment, and uses the physics outcomes to inform its answer. This approach allows models that are 100x smaller to outperform massive vanilla text-only models on physical reasoning tasks.
By moving from pure statistical text prediction to multimodal integration and interactive simulation, the field is shifting from building surface-level pattern matchers to creating systems that can ground abstract symbols in the constraints of the physical world.
Key Foundational Work
Perceptually Grounded Selectional Preferences (2015) — contributed 1 concept elements to this direction.
Affordance Extraction and Inference based on Semantic Role Labeling (2018) — contributed 1 concept elements to this direction.