TL;DR
This paper introduces visually grounded word embeddings (VGEs) by combining text and images, demonstrating they better capture human semantic knowledge than text-only embeddings. VGEs show improved prediction of human reaction times in semantic priming tasks and correlate better with human similarity judgments, even when compared to text embeddings trained on massive corpora.
What Problem Does This Paper Address?
Traditional word embeddings, like Word2Vec or GloVe, learn word meanings based solely on their co-occurrence in text. While powerful, this approach has limitations: it requires vast amounts of text data (billions of words) to achieve high quality, which is cognitively implausible for humans. Furthermore, human understanding of word meaning is deeply rooted in sensory experiences (seeing, hearing, touching), which are not fully captured by text alone. This paper addresses the gap by exploring whether incorporating visual information into word embeddings can lead to representations that are more cognitively plausible and better reflect human semantic knowledge.
Methods
The core of this work is the creation of Visually Grounded Embeddings (VGEs). This is achieved using an end-to-end approach that combines image and text data. Specifically, they leverage a caption-image retrieval (C2I) model. The model consists of:
- Image Embedder: A ResNet-152 network extracts visual features from images. These features are then passed through a linear projection layer.
- Caption Embedder: This part uses a word embedding layer followed by a two-layer bidirectional LSTM with a self-attention mechanism to create a contextualized representation of the text.
The VGE model is trained to make embeddings for matching image-caption pairs similar and mismatched pairs dissimilar. Crucially, the word embeddings used for VGEs are derived from the hidden activations of the LSTM layer, not just the input embedding layer, as these are found to better capture semantic meaning.
For comparison, they also train text-based word embeddings (Word2Vec, FastText, GloVe) on the same MSCOCO caption data. Additionally, they use publicly available pretrained text embeddings trained on much larger corpora.
Datasets
- Training Data: The MSCOCO dataset is used for training both the VGE model and the custom text-based embeddings. MSCOCO contains over 123,000 images with multiple English captions per image.
- Evaluation Data:
- Semantic Similarity Judgements: Several datasets are used, including WordSim353, WordSim-S, WordSim-R, SimLex999 (and its subsets Q1, Q4), MEN, and Rare Words. These datasets contain word pairs annotated by humans for similarity or relatedness.
- Semantic Priming: The Semantic Priming Project (SPP) dataset is used, which contains lexical decision and naming times from a large priming experiment.
Key Insights and Ideas
- Visual grounding enhances semantic representation: The core idea is that human semantic knowledge is embodied and multimodal. By grounding word embeddings in visual information, they can capture aspects of meaning that text alone misses.
- Cognitive plausibility matters: The paper emphasizes creating embeddings from a "realistic amount of linguistic exposure," contrasting with the massive corpora used for many state-of-the-art text embeddings.
- Hidden states are key: Deriving word embeddings from the hidden activations of the LSTM in the caption embedder, rather than just the input layer, proved more effective for capturing semantic meaning.
- Complementary information: VGEs capture unique variance in human judgments, suggesting they provide information that is complementary to, not just redundant with, text-based embeddings.
- Not all words benefit equally: The effectiveness of VGEs might depend on the word's nature. They perform particularly well on concrete, picturable nouns but less so on abstract or rare words.
Key Results
- Semantic Similarity:
- VGEs trained on MSCOCO performed comparably to text-based methods trained on the same corpus.
- VGEs explained a significant portion of unique variance in human similarity judgments, even after controlling for text-based embeddings (both custom-trained and large-scale pretrained). For example, in Figure 1, the grey bars (semi-partial R²) show substantial contributions from VGEs across most datasets.
- VGEs showed particular strength on datasets with concrete, picturable nouns (e.g., SimLex999 Q4).
- Semantic Priming:
- VGEs significantly improved regression models predicting human reaction times in the SPP dataset, outperforming text-based models.
- The positive impact of VGEs was observed across both short and long stimulus onset asynchrony (SOA) durations, suggesting visual information is not limited to early processing stages.
- VGEs showed significant interactions with Task (lexical decision vs. naming), indicating their effect is stronger in tasks more sensitive to semantic processing.
Limitations and Future Work
- Rare Words: VGEs performed notably worse on the Rare Words dataset. The authors attribute this to the fact that VGEs are grounded in image context, and if words in a sentence are not visually represented or are abstract, the visual grounding may not be helpful or could even be counterproductive.
- Strong Embodied Cognition Theory: The results do not fully support the strongest formulations of embodied cognition theory that suggest total equivalence between conceptual and sensorimotor processing. The authors suggest that while visual grounding is important, it might not play a role for all types of words (e.g., abstract ones).
- Future Work: The paper implicitly suggests further exploration into how different types of visual information (beyond static images) or different modalities could further enhance word embeddings. Investigating the specific types of semantic knowledge captured by VGEs versus text-based models could also be a fruitful direction.
Summary updated