Introduction
Can a machine that has never felt the pull of gravity or the solidness of a floor truly understand concepts like "up" or "support"? This research investigates whether Large Language Models (LLMs) capture human-like spatial intuitions by replicating three classic psycholinguistic experiments. The findings reveal that while models like GPT-4 correlate remarkably well with human spatial metaphors, Vision-Language Models (VLMs) often struggle to bridge the gap between abstract concepts and visual spatial representations.
The Symbol Grounding Problem
At the core of this study is the Symbol Grounding Problem: can symbols (words) acquire meaning solely through their relationships with other symbols, or do they require "grounding" in physical, sensory experience?
Humans develop "image schemas"—fundamental cognitive building blocks such as VERTICALITY, SUPPORT, or BALANCE—through early sensorimotor experiences. Cognitive linguistics suggests we reuse these spatial schemas to understand abstract language (e.g., "supporting a friend" or "feeling down"). Since LLMs are non-embodied, researchers sought to determine if these models can "intuitionize" the spatial foundations of language or if they are merely performing sophisticated statistical mimicry.
Methodology
The authors reproduced three landmark psycholinguistic experiments using a suite of modern models, including LLaMA-2 (13b and 70b), GPT-3.5, GPT-4, and Vision-Language Models like IDEFICS and GPT-4vision.
- Experiment 1 (Gibbs et al., 1994): Models rated how strongly various uses of the verb "stand" (e.g., "stand at attention") related to specific image schemas like VERTICALITY on a 1–7 scale.
- Experiment 2 (Beitel et al., 2001): Focused on the polysemy of the preposition "on" (e.g., "the fly is on the ceiling" vs. "the bus is on schedule").
- Experiment 3 (Richardson et al., 2001): Models associated 30 verbs (e.g., "lifted", "respected") with spatial directions (Up, Down, Left, Right) across textual, pseudo-visual (Unicode arrows), and visual modalities.
Key Insights and Results
- Abstract Alignment: GPT-4 showed a moderate-to-strong correlation (~0.61) with human intuitions, suggesting that the "spatiality" of human thought is deeply embedded in the structure of text.
- The VLM Paradox: Surprisingly, VLMs performed worse than text-only models on some spatial tasks. Open-source VLMs like IDEFICS failed to interpret abstract line drawings effectively, showing near-zero correlation in visual directional tasks.
- Emergent Intuition: Performance improved significantly with model scale and instruction tuning, with a clear gap between LLaMA-13b and GPT-4.
As shown in the table above, GPT-4 achieved the highest similarity to humans, with Spearman correlations reaching 0.69 for VERTICALITY and 0.82 for RESISTANCE in Experiment 1. In Experiment 3, while GPT-4 maintained strong textual correlation (~0.70), the open-source IDEFICS model showed 0.00 correlation in the visual condition.
Limitations and Future Work
The study notes several constraints, including the narrow demographic of the original human data (US undergraduates) and potential "data contamination" since GPT-4 is proprietary. Future work aims to expand to multilingual models and more complex visual stimuli to further probe the boundaries of machine spatial reasoning.
Summary updated