Introduction
While Large Language Models (LLMs) demonstrate remarkable linguistic capabilities, their actual "understanding" of the physical world remains a subject of intense research. Humans possess a vast reservoir of common sense—knowing, for instance, that a toothbrush belongs in a bathroom rather than a kitchen. However, because such facts are obvious, they are rarely explicitly stated in text, a phenomenon known as reporting bias. This paper investigates the extent to which modern AI models encode these implicit relationships between objects, their parts, and their typical environments.
Methodology
The researchers evaluated two distinct generations of AI models:
- Static Models: Older word embedding techniques (e.g., Word2Vec, GloVe, and fastText) where each word is represented by a single, fixed vector.
- Contextualized Models (Transformers): Modern architectures (e.g., BERT, RoBERTa, and GPT) where word representations shift based on surrounding context.
To extract knowledge from these models, three "probes" were used: simple mathematical similarity (Cosine Similarity), mask-filling tasks (asking the model to predict a missing word in a sentence), and supervised classifiers (training small Feed-Forward Networks, or FFNs, to predict relationships from the models' internal data).
These probes were tested against three ground-truth datasets: NYU Depth V2 for spatial containment, Online-Bildwörterbuch for part-whole relations, and HowToKB for action-object associations.
Key Insights: The Probing Gap and the Static Surprise
A primary finding of the study is the "Probing Gap." The method used to query a model matters as much as the model's architecture. While simple similarity measures performed poorly, the use of a dedicated FFN classifier caused accuracy to skyrocket—with performance gains between 6% and 52%. This suggests that common-sense knowledge is present within these models but is encoded in complex, non-linear ways that simple metrics cannot capture.
Surprisingly, older static models like GloVe and fastText often performed as well as, or even better than, massive Transformers like BERT. This is particularly evident in part-whole relationship tasks. Static models aggregate global word co-occurrence statistics, which may capture general "common sense" more robustly than the context-heavy, specific representations generated by Transformers.
Model Scaling and Spatial Clarity
The study also revealed that bigger is not always better. BERT-Base consistently outperformed the much larger BERT-Large, suggesting that increased parameter counts may introduce noise or make specific associations harder to retrieve.
Furthermore, the models exhibited varying levels of "spatial clarity." They were most accurate at identifying objects in bathrooms and kitchens—rooms defined by highly specific artifacts like toilets or stoves. Conversely, they struggled with living rooms and offices, where objects like chairs or televisions are more generic and appear across multiple room types.
Conclusion
This research highlights that while Transformers do encode a structured view of the world, this knowledge is often buried. The fact that older, simpler models can rival modern LLMs in common-sense tasks suggests that architectural complexity is not the only path to world understanding. Future work will explore grounded models—those trained on both text and images—to see if visual experience helps AI better grasp the physical structure of our world.
Summary updated