Abstract
When people think of everyday things like an egg, they typically have a mental image associated with it. This allows them to correctly judge, for example, that "the yolk surrounds the shell" is a false statement. Do language models similarly have a coherent picture of such everyday things? To investigate this, we propose a benchmark dataset consisting of 100 everyday things, their parts, and the relationships between these parts, expressed as 11,720 "X relation Y?" true/false questions. Using these questions as probes, we observe that state-of-the-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent "parts mental models" (54-59% accurate, 19-43% conditional constraint violation). We propose an extension where we add a constraint satisfaction layer on top of the LM's raw predictions to apply commonsense constraints. As well as removing inconsistencies, we find that this also significantly improves accuracy (by 16-20%), suggesting how the incoherence of the LM's pictures of everyday things can be significantly reduced.
Introduction
When a human conceptualizes an everyday object, they don't merely recall a list of attributes; they possess a "mental model"—a structured representation of the object's components and their spatial or functional relationships. This internal map allows us to immediately identify logical fallacies, such as the claim that an egg's yolk surrounds its shell.
Large Language Models (LLMs) are often critiqued as "stochastic parrots" that predict text based on statistical patterns without underlying understanding. This research investigates whether LLMs like GPT-3 possess coherent internal representations of physical objects or if they merely recite disconnected fragments of knowledge.
The ParRoT Benchmark
To probe these internal representations, we introduced the ParRoT (Parts and Relations of Things) benchmark. It encompasses:
- 100 Everyday Objects: Ranging from natural entities like trees to complex man-made devices like flashlights and bicycles.
- 2,200 Unique Parts: Identified across the object set.
- 11,720 Relationship Tuples: Covering 14 distinct relation types (e.g., surrounds, inside, above, connects).
- Ground Truth: Established via human crowdworkers who drew "parts graphs" for each object, providing a gold standard for spatial and functional coherence.
Methodology: ParRoT-Con
We developed a two-stage approach, ParRoT-Con, to extract and refine the mental models of LLMs:
- Probing: The model is queried with True/False questions regarding every possible relationship between an object's parts. The model's confidence (probability) for each assertion is recorded.
- Constraint Reasoning: To address the inherent incoherence in raw LM outputs, we apply a neuro-symbolic layer. We define four categories of commonsense constraints: symmetric, asymmetric, inverse, and transitive. A Weighted MaxSAT solver identifies the most probable set of relationships that satisfies all these logical constraints, effectively forcing the model into a state of global consistency.
Results and Discussion
Our analysis reveals a significant "incoherence gap" in base LLMs. Models frequently generate contradictory beliefs, such as asserting that an egg white both surrounds and is surrounded by the shell. GPT-3 and Macaw exhibited conditional constraint violations between 19% and 43% of the time.
While base models achieved only 54-59% accuracy—barely outperforming random chance—the addition of the constraint reasoning layer yielded a dramatic improvement. Accuracy rose to 70-79%, and logical violations were reduced to 0%. This suggests that while LLMs possess the necessary "fragments" of knowledge within their weights, they lack the intrinsic reasoning mechanisms to organize that knowledge into a coherent whole. Interestingly, models performed better on natural objects (animals, trees) than on man-made tools, likely due to more standardized spatial descriptions of biological entities in training corpora.
Limitations and Future Work
While the ParRoT-Con approach significantly improves coherence, several limitations remain. The current study focuses on 14 spatial and functional relations, excluding properties like material or size. Additionally, the benchmark represents a static snapshot; as technology evolves, human mental models of objects change. Future research will investigate the generalizability of this neuro-symbolic approach to other domains, such as temporal timelines or geographical reasoning.
Conclusion
This work demonstrates that the incoherence of LLMs regarding the physical world can be significantly mitigated by combining their broad knowledge with symbolic reasoning. By enforcing commonsense constraints, we can transform fragmented predictions into logically sound mental models, bridging the gap between statistical text prediction and coherent physical reasoning.
Summary updated