Visual grounding of abstract and concrete words: A response to Günther et al. (2020)

Hassan Shahmohammadi, Maria Heitmeier, Elnaz Shafaei-Bajestan, Hendrik P. A. Lensch, Harald Baayen

TL;DR

This research challenges the "embodied cognition" hypothesis, which posits that humans rely on mental imagery to ground the meaning of words. By re-evaluating a key behavioral study, the authors show that simple text-based associations—matching words to labels of objects within a picture—can explain human performance as effectively as complex visual grounding models.

Introduction: The Symbol Grounding Problem

A central question in cognitive science is the "symbol grounding problem": how do abstract symbols (words) acquire meaning if they only refer to other symbols? The grounded cognition theory suggests that word meanings are rooted in sensory and motor experiences. For instance, the word "apple" is understood through the mental simulation of its visual, olfactory, and gustatory properties.

Günther et al. (2020) recently claimed to demonstrate a direct link between words and embodied experience, showing that humans could match abstract words (e.g., "childhood") to images predicted by computational models. However, Shahmohammadi et al. argue that this may be an overinterpretation, suggesting that participants might simply be performing rapid word associations based on visible objects in the images.

Methodology

To test their hypothesis, the authors employed computational methods that do not rely on internal mental simulation to solve the original behavioral task.

1. The "Max" Model

This lean baseline model uses a pre-trained Convolutional Neural Network (CNN) to identify the top 10 objects in an image. It then calculates the cosine similarity between the target word's embedding and the labels of these objects. If the target word is more semantically similar to the labels in one image over another, the model selects that image.

2. Zero-Shot Grounding (ZSG)

The authors compared standard textual embeddings (GloVe and Word2Vec) against their own ZSG embeddings. These embeddings are specifically trained to align text with visual features extracted from the MS-COCO dataset.

The Zero-Shot Grounding (ZSG) architecture for mapping word embeddings to visual feature vectors.

3. Generalized Additive Models (GAMs)

Statistical GAMs were used to identify which factors—such as the number of objects or the similarity of object labels—best predicted human choices in the behavioral experiments.

Key Results

The study's findings suggest that visual grounding may play a smaller role in these tasks than previously thought:

  • Human-Level Performance without Vision: The simple "Max" model, relying solely on text-based similarities to object labels, achieved approximately 70% accuracy, matching the mean performance of human participants.
  • Grounded vs. Textual Embeddings: Visually grounded ZSG embeddings provided only a marginal (1-2%) improvement over purely textual ones in the "prototype" setup and performed worse in the "exemplar" setup.
  • Predictors of Choice: GAM analysis revealed that "Predicted Image Similarity" (based on object labels) was a dominant predictor of human choice, supporting the theory that humans scan for recognizable objects rather than comparing images to holistic mental representations.
Comparison of Prototype and Exemplar model architectures for training and retrieval.

Discussion: Semantic vs. Visual Space

The "aha moment" of this research is the realization that a "mental picture" of an abstract concept like "childhood" is unnecessary for image matching. Instead, one only needs to recognize "children" in a photo and know that the word "children" is semantically close to "childhood" in a text-based vector space. This suggests that a vast amount of "visual" knowledge is actually encoded within our linguistic patterns.

For abstract words, grounding appears to be indirect. While we cannot "see" jealousy, we see the concrete objects and situations associated with it. The visual alignment of concrete words (like "horse") helps "pull" the abstract words they are associated with into a more precise semantic position.

Conclusion

While these findings do not entirely disprove embodied cognition, they suggest that the experimental design used by Günther et al. is a "noisy" measure of visual grounding because it can be solved via simple linguistic associations. The authors call for more sophisticated experiments that isolate visual information that cannot be easily labeled, and suggest exploring the role of emotional valence in grounding abstract concepts.

Summary updated

BibTeX
@inproceedings{shahmohammadi2023visual,
  author = {Hassan Shahmohammadi and Maria Heitmeier and Elnaz Shafaei-Bajestan and Hendrik P. A. Lensch and Harald Baayen},
  title = {{Visual grounding of abstract and concrete words: A response to Günther et al. (2020)}},
  booktitle = {The Mental Lexicon},
  year = {2023},
  doi = {10.1075/ml.22010.sha},
}
Figure