How direct is the link between words and images?

Hassan Shahmohammadi, Maria Heitmeier, Elnaz Shafaei-Bajestan, Hendrik P. A. Lensch, R. Harald Baayen

TL;DR

This paper challenges the idea that humans rely on "embodied simulation" (mental imagery) to match words with images. By demonstrating that purely text-based models can predict human choices nearly as well as visually grounded ones, the authors argue that these tasks are likely solved through simple semantic associations between words and object labels rather than a direct "perceptual" link.

What Problem Does This Paper Address?

In cognitive science, the Symbol Grounding Problem asks how abstract linguistic symbols (words) get their meaning from the real world. While traditional AI models like Word2Vec learn meaning purely from text co-occurrence, recent research by Günther et al. (2022) suggested that humans possess a direct link to "embodied experience." They showed that humans could successfully pick a model-predicted image for a word (like "stallion" or even "childhood") over a random one.

The authors of this paper question that conclusion. They ask: Are humans actually generating mental images to solve these tasks, or are they just using a "shortcut"? Specifically, they investigate if the task can be solved by simply identifying objects in the pictures and checking their textual similarity to the target word.

Methods

The researchers introduced a "task analysis" approach to model human behavior using two main strategies:

  1. The "Max" Model: Instead of treating an image as a holistic visual blob, they used a Convolutional Neural Network (CNN) to extract the names of the top 10 objects visible in the image. They then calculated the cosine similarity between the target word's embedding and the embeddings of these object labels. If the word "childhood" is more similar to the label "children" than to "plant," the model picks the image containing children.
  2. Comparison of Embeddings: They compared purely textual embeddings (GloVe and Word2Vec) against Zero-shot Grounded (ZSG) embeddings. ZSG embeddings are "informed" by vision because they are trained to align text vectors with image features from the MS-COCO dataset.
  3. Generalized Additive Models (GAMs): They used these statistical models to determine which factors (like the number of objects or the similarity scores) best predicted the actual decisions made by human participants in previous studies.

Datasets

  • MS-COCO 2017: Used to train the ZSG alignment between text and images.
  • ImageNet: The source of images used in the original behavioral experiments.
  • GPVM Behavioral Data: The original response data from 110 human participants across two experiments by Günther et al. (2022).
  • Standard Benchmarks: Evaluation also referenced common word similarity datasets like MEN and SimLex999.

Key Insights and Ideas

  • The "Labeling" Strategy: The most striking insight is that humans likely solve word-image matching by identifying objects. If you see a picture of a horse and the word is "stallion," you don't need a complex "embodied simulation"—you just recognize a "horse" and know it's related to "stallion."
  • Text is (Mostly) Enough: Purely text-based models (GloVe) were surprisingly effective at predicting human choices. This suggests that much of what we think is "visual knowledge" is actually already encoded in the way we talk about things.
  • Grounding’s Modest Gain: While visually grounded embeddings did show a slight numerical advantage in some cases (particularly for concrete words), the improvement was often statistically insignificant.
  • Abstract Words and Context: For abstract words like "jealousy," the link to vision is indirect. The paper suggests that grounding helps here not by "showing" jealousy, but by refining the semantic space of the concrete words that abstract concepts are built upon (Conceptual Metaphor Theory).

Key Results

  • Virtual Participants: The "Max" models (using only text-based similarity to object labels) achieved an accuracy of ~70% in predicting which image humans would choose—nearly identical to the actual human mean.
  • Grounded vs. Textual: In the "Prototype" model setup, grounded GloVe improved accuracy by only 2% over standard GloVe. In the "Exemplar" setup, grounding actually decreased accuracy by 4%.
  • Predictors of Choice: The GAMs showed that "Predicted Image Similarity" (how well the objects in the image matched the word) was a massive predictor of human choice, supporting the theory that humans are scanning for recognizable objects rather than processing images holistically.

Limitations and Future Work

  • Visual Only: The study focuses exclusively on the link between words and vision. Human experience is multimodal, involving sound, touch, and emotion, which are not captured here.
  • CNN Limitations: The object detection relies on a pre-trained CNN. If the CNN misses a subtle visual cue that a human would catch, the model's similarity score might be off.
  • The "How" of Integration: The authors note that while they've shown text can solve the task, the real challenge for future work is understanding exactly how humans combine linguistic and visual information in real-time to form meaning.
  • Emotion: Future research could explore if abstract words are grounded more in emotional valence than in visual objects.

Summary updated

Abstract

Abstract Günther et al. (2022) investigated the relationship between words and images in which they concluded the possibility of a direct link between words and embodied experience. In their study, participants were presented with a target noun and a pair of images, one chosen by their model and another chosen randomly. Participants were asked to select the image that best matched the target noun. Building upon their work, we addressed the following questions. 1. Apart from utilizing visually embodied simulation, what other strategies subjects might have used? How much does this setup rely on visual information? Can it be solved using textual representations? 2. Do current visually-grounded embeddings explain subjects’ selection behavior better than textual embeddings? 3. Does visual grounding improve the representations of both concrete and abstract words? For this aim, we designed novel experiments based on pre-trained word embeddings. Our experiments reveal that subjects’ selection behavior is explained to a large extend on text-based embeddings and word-based similarities. Visually grounded embeddings offered modest advantages over textual embeddings in certain cases. These findings indicate that the experiment by Günther et al. (2022) may not be well suited for tapping into the perceptual experience of participants, and the extent to which it measures visually grounded knowledge is unclear.
BibTeX
@inproceedings{shahmohammadi2023how,
  author = {Hassan Shahmohammadi and Maria Heitmeier and Elnaz Shafaei-Bajestan and Hendrik P. A. Lensch and R. Harald Baayen},
  title = {{How direct is the link between words and images?}},
  booktitle = {The Mental Lexicon},
  year = {2023},
  doi = {10.1075/ml.22010.sha},
}
Figure