Abstract
Current word embedding models, despite their success, still suffer from their lack of grounding in the real world. In this line of research, Günther et al. (2022) proposed a behavioral experiment to investigate the relationship between words and images. In their setup, participants were presented with a target noun and a pair of images, one chosen by their model and another chosen randomly. Participants were asked to select the image that best matched the target noun. In most cases, participants preferred the image selected by the model. Günther et al., therefore, concluded the possibility of a direct link between words and embodied experience. We took their experiment as a point of departure and addressed the following questions:
- Apart from utilizing visually embodied simulation of given images, what other strategies might subjects have used to solve this task? To what extent does this setup rely on visual information from images? Can it be solved using purely textual representations?
- Do current visually grounded embeddings explain subjects' selection behavior better than textual embeddings?
- Does visual grounding improve the semantic representations of both concrete and abstract words? To address these questions, we designed novel experiments by using pre-trained textual and visually grounded word embeddings. Our experiments reveal that subjects' selection behavior is explained to a large extent based on purely text-based embeddings and word-based similarities, suggesting a minor involvement of active embodied experiences. Visually grounded embeddings offered modest advantages over textual embeddings only in certain cases. These findings indicate that the experiment by Günther et al. may not be well suited for tapping into the perceptual experience of participants, and therefore the extent to which it measures visually grounded knowledge is unclear.
Introduction: The Symbol Grounding Problem
In cognitive science, the Symbol Grounding Problem asks how abstract linguistic symbols (words) acquire meaning from the physical world. While traditional AI models like Word2Vec learn meaning purely from text co-occurrence, recent research by Günther et al. (2022) suggested that humans possess a direct link to "embodied experience." They demonstrated that humans could successfully pick a model-predicted image for a word (like "stallion" or even "childhood") over a random one, implying a form of mental imagery or simulation.
This paper challenges that conclusion. We investigate whether humans are actually generating mental images to solve these tasks, or if they are using a "shortcut"—specifically, identifying objects in pictures and checking their textual similarity to the target word.
Methodology: Task Analysis and Model Architectures
To model human behavior, we introduced a task analysis approach using two main strategies to compare purely textual embeddings (GloVe and Word2Vec) against Zero-shot Grounded (ZSG) embeddings. ZSG embeddings are informed by vision because they align text vectors with image features from the MS-COCO dataset.
- The "Max" Model: Instead of treating an image as a holistic visual blob, we used a Convolutional Neural Network (CNN) to extract the names of the top 10 objects visible in the image. We then calculated the cosine similarity between the target word's embedding and the embeddings of these object labels.
- Prototype vs. Exemplar Models: We evaluated different ways of representing the relationship between words and visual data. The Prototype model represents a word by the average vector of its associated training images, while the Exemplar model stores individual instances.
Key Insights: The Power of Textual Associations
Our analysis yielded several striking insights regarding how humans likely solve word-image matching:
- The "Labeling" Strategy: Humans likely solve these tasks by identifying objects. If you see a picture of a horse and the word is "stallion," you don't need a complex "embodied simulation"—you simply recognize a "horse" and know it is semantically related to "stallion."
- Text is (Mostly) Enough: Purely text-based models (GloVe) were surprisingly effective at predicting human choices. This suggests that much of what we consider "visual knowledge" is already encoded in the statistical structure of language.
- Grounding’s Modest Gain: While visually grounded embeddings showed a slight numerical advantage for concrete words, the improvement was often statistically insignificant.
- Abstract Words: For abstract words like "jealousy," the link to vision is indirect. Grounding may help refine the semantic space of the concrete words that abstract concepts are built upon, rather than providing a direct visual representation of the abstract concept itself.
Results: Evaluating the Models
We used Generalized Additive Models (GAMs) to determine which factors best predicted the actual decisions made by human participants in the original Günther et al. studies.
- Virtual Participants: The "Max" models (using only text-based similarity to object labels) achieved an accuracy of ~70% in predicting human choices—nearly identical to the actual human mean.
- Grounded vs. Textual Performance: In the Prototype model setup, grounded GloVe improved accuracy by only 2% over standard GloVe. Conversely, in the Exemplar setup, grounding actually decreased accuracy by 4%.
The GAMs showed that "Predicted Image Similarity" (how well the objects in the image matched the word) was a massive predictor of human choice, supporting the theory that humans scan for recognizable objects rather than processing images holistically.
Conclusion
Our findings indicate that the experiment by Günther et al. may not be well-suited for tapping into the perceptual experience of participants. Because text-based models can solve the task with high accuracy, the extent to which this setup measures visually grounded knowledge remains unclear. While we have shown that text can solve the task, the real challenge for future work is understanding exactly how humans combine linguistic and visual information in real-time to form meaning, potentially involving emotional valence or multimodal integration beyond simple vision.
Summary updated