Do language models have coherent mental models of everyday things?

Yuling Gu, Bhavana Dalvi Mishra, Peter Clark

Introduction

When a human thinks of an egg, they possess a structured mental image: the yolk is inside the white, and both are encased in a shell. This internal representation allows us to immediately reject a statement like "the yolk surrounds the shell." While Large Language Models (LLMs) are trained on vast amounts of text and can generate fluent descriptions of objects, it remains unclear if they develop similar coherent "mental models" or if they merely possess "fragments of knowledge."

Figure 2: Comparison of human mental models (A), incoherent LM beliefs (B), and the corrected coherent model (C) achieved through the ParRoT-Con framework.

Previous research has shown that LMs struggle with physical commonsense. This paper specifically investigates coherence: if an LM believes "A is above B," does it also consistently believe "B is below A"? The authors argue that without such coherence, LMs cannot truly reason about or interact with the physical world effectively.

Methodology: The ParRoT-Con Framework

To address this incoherence, the researchers developed ParRoT-Con (Parts and Relations of Things with Constraints), a two-step neuro-symbolic approach:

  1. Probing: The system queries a pre-trained LM (such as GPT-3 or Macaw) with an exhaustive list of True/False questions regarding the relationships between parts of an object (e.g., "In an egg, does the yolk surround the shell?"). This generates a set of "raw" beliefs with associated confidence scores.
  2. Constraint Reasoning: A neuro-symbolic layer is added on top of the LM's raw predictions. This layer uses a MaxSAT (Maximum Satisfiability) solver to enforce four types of commonsense constraints: symmetry (next-to), asymmetry (part-of), inversion (above/below), and transitivity (inside/inside).

The solver finds the most probable set of beliefs that satisfies all hard constraints, effectively "self-correcting" the LM's logical inconsistencies.

The ParRoT Benchmark

The authors introduced a new benchmark dataset to quantify these mental models:

  • ParRoT: Consists of 100 everyday things, ranging from natural objects like "trees" to man-made devices like "flashlights." It includes 2,191 parts and 11,720 relationship tuples.
  • Gold Standard: To create the ground truth, human annotators drew "parts graphs" for each object, which were then converted into relationship tuples for model evaluation.
Figure 5: Examples of objects from the ParRoT dataset (Tree and Flashlight) and their corresponding human-annotated parts graphs used for evaluation.

Key Insights and Results

The study revealed several critical findings regarding the current state of LM world-knowledge:

  • The "Stochastic Parrot" Reality: LMs are surprisingly poor at spatial reasoning. Base models achieved only 54-59% accuracy on ParRoT—barely outperforming a majority-class baseline.
  • Prevalent Incoherence: Models frequently contradict themselves. For example, GPT-3 might claim that a yolk surrounds a shell and vice versa. Conditional constraint violations ranged from 19% to 43%.
  • Accuracy Boost via Reasoning: Applying the constraint satisfaction layer (ParRoT-Con) improved the accuracy of mental models by 16-20% and reduced the conditional constraint violation rate to 0%.
  • Natural vs. Man-made: LMs generally have better mental models of natural living things (animals, plants) than complex man-made devices (typewriters, computers), likely due to more consistent descriptions in training data.

Conclusion

The results suggest that simply making models larger may not solve the coherence problem. Instead, a hybrid architecture that combines the broad knowledge of an LM with the rigorous logic of a symbolic reasoner is a more promising path toward developing robust "world models." While ParRoT-Con significantly closes the gap, a substantial performance difference remains between LMs and human-level coherence, highlighting the ongoing challenge of teaching machines to understand the physical world.

Summary updated

Abstract

When people think of everyday things like an egg, they typically have a mental image associated with it. This allows them to correctly judge, for example, that "the yolk surrounds the shell" is a false statement. Do language models similarly have a coherent picture of such everyday things? To investigate this, we propose a benchmark dataset consisting of 100 everyday things, their parts, and the relationships between these parts, expressed as 11,720 "X relation Y?" true/false questions. Using these questions as probes, we observe that state-of-the-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent "parts mental models" (54-59% accurate, 19-43% conditional constraint violation). We propose an extension where we add a constraint satisfaction layer on top of the LM's raw predictions to apply commonsense constraints. As well as removing inconsistencies, we find that this also significantly improves accuracy (by 16-20%), suggesting how the incoherence of the LM's pictures of everyday things can be significantly reduced.
BibTeX
@inproceedings{gu2023do,
  author = {Yuling Gu and Bhavana Dalvi Mishra and Peter Clark},
  title = {{Do language models have coherent mental models of everyday things?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=qZoUJIXz5Xd},
}
Figure