Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?

Zhihui Yang, Yupei Wang, Kaijie Mo, Zhe Zhao, Renfen Hu

Introduction

Large Language Models (LLMs) have demonstrated remarkable proficiency in text generation and reasoning. However, a persistent critique is their lack of "grounding"—a genuine understanding of the physical properties and sensory experiences that underpin human language. While Vision-Language Models (VLMs) are trained on multimodal data to bridge this gap, it remains unclear if "seeing" objects translates to a deeper grasp of embodied knowledge. This research investigates whether visual grounding actually helps models understand the physical world or if they merely learn surface-level statistical associations.

Methodology: Probing the Embodied Mind

To evaluate the "embodied" capabilities of 30 state-of-the-art models, researchers developed a two-pronged benchmark: SensoryVec and PerceptualQA.

Overview of the SensoryVec and PerceptualQA evaluation framework.

1. SensoryVec

This task assesses internal vector representations (embeddings). By analyzing 349 groups of sensory adjectives (e.g., small, little, big), the researchers tested whether a model's latent space correctly clusters synonyms closer than antonyms. This reveals if the model's "mental map" of sensory concepts aligns with physical reality.

2. PerceptualQA

PerceptualQA consists of 1,400 multiple-choice questions requiring "embodied imagination." Crucially, these questions are presented as text-only during inference. The model must rely on its internal understanding of sensory experiences—such as spatial orientation or tactile properties—to answer correctly. For instance, a model might be asked to predict the orientation of a rotated character or the relative height of body parts in specific poses.

Benchmarking Sensory Knowledge

The benchmark covers a wide spectrum of human perception, including visual, auditory, tactile, gustatory, olfactory, and interoceptive senses. The PerceptualQA component is particularly detailed in its visual subtasks, which range from color attributes to complex geometric transformations.

Example questions from the PerceptualQA benchmark across various sensory tasks.

To establish a performance ceiling, the researchers established a human baseline using graduate students, who achieved an average accuracy of 86.00%.

Key Insights and Results

The study yielded several surprising findings that challenge the assumption that more modalities lead to better physical understanding:

  • The VLM Paradox: VLMs do not consistently outperform text-only models in embodied knowledge tasks. In the SensoryVec task, the text-only BERT-base model achieved the highest accuracy (72.21%), surpassing much larger multimodal counterparts.
  • The Spatial Reasoning Barrier: Models struggle significantly with tasks involving geometry, symbols, and body mechanics. While they perform relatively well on taste or smell (likely due to strong textual associations), they fail at "mental rotation" and complex positional reasoning.
  • Top Performers: Claude 3.5 Sonnet emerged as the leading model in PerceptualQA with 69.04% accuracy, though it remains significantly behind the human baseline.
  • Marginal Multimodal Gains: When comparing VLMs to their direct text-only ancestors (e.g., Qwen-VL vs. Qwen), the addition of visual training only resulted in a marginal 2.32% accuracy improvement on PerceptualQA.
  • Form over Substance: Model representations are often dominated by "word form" and frequency bias. Models frequently mistake physical opposites for synonyms if the words appear in similar textual contexts or share linguistic roots.

Discussion: Beyond Static Training

The results suggest that current visual grounding methods—primarily training on static image-text pairs—capture surface-level features but fail to instill the deep, causal logic of the physical world. Humans learn these concepts through active interaction and temporal observation, which static datasets cannot replicate.

To overcome these limitations, future research should explore:

  • Dynamic Data: Incorporating video sequences to teach temporal and causal relationships.
  • Active Exploration: Allowing models to interact with simulated environments to learn through feedback.
  • Diverse Sensors: Integrating non-visual data like haptic feedback or motion capture to broaden the model's sensory horizon.

Ultimately, achieving true embodied intelligence in AI may require moving beyond passive observation toward active, multi-sensory engagement with the world.

Summary updated

BibTeX
@inproceedings{yang2025does,
  author = {Zhihui Yang and Yupei Wang and Kaijie Mo and Zhe Zhao and Renfen Hu},
  title = {{Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure