Abstract
The question of how meaning might be acquired by young children and represented by adult speakers of a language is one of the most debated topics in cognitive science. Existing semantic representation models are primarily amodal based on information provided by the linguistic input despite ample evidence indicating that the cognitive system is also sensitive to perceptual information. In this work we exploit the vast resource of images and associated documents available on the web and develop a model of multimodal meaning representation which is based on the linguistic and visual context. Experimental results show that a closer correspondence to human data can be obtained by taking the visual modality into account.
Introduction
Traditional computational models of semantics, such as Latent Semantic Analysis (LSA), typically rely on the distributional hypothesis: the idea that words appearing in similar linguistic contexts share similar meanings. However, these "amodal" models ignore a crucial aspect of human cognition—our language is grounded in a physical, visual world. Humans do not learn language from text alone; we learn through a rich stream of perceptual evidence. This paper addresses the gap in cognitive realism by proposing a model that integrates discrete linguistic data with high-dimensional visual information.
Methodology: Multimodal Topic Modeling
The core of this research is the development of a multimodal Latent Dirichlet Allocation (MixLDA) model. To bridge the gap between text and images, the authors treat visual features as a discrete vocabulary of "visual words" (visiterms).
Image Quantization and Representation
Images are converted into a format compatible with text-based topic models through quantization. The researchers explored several methods for extracting visual features, including Scale Invariant Feature Transform (SIFT) to detect local interest points, grid-based sampling, and image segmentation.
Once features are extracted, they are clustered using K-means to create a visual vocabulary. Each document in the training corpus is then represented as a "Bag-of-Mixed-Words," containing both standard textual tokens and these newly defined visual words.
The MixLDA Model
The LDA framework is applied to this mixed collection. In MixLDA, a latent "topic" represents a probability distribution over both the textual and visual vocabularies. This allows the model to capture cross-modal associations, such as the visual texture of grass being associated with the word "sports" or "football."
Experimental Results
The model was trained on a corpus of 3,361 documents from the BBC News website, where each article was paired with its original image. Performance was evaluated against human gold standards: the Nelson et al. (1999) word association norms and the WordSim353 similarity dataset.
Performance Comparison
The multimodal approach (MixLDA) consistently outperformed the text-only baseline (TxtLDA). For word association tasks, MixLDA achieved a Pearson’s correlation of 0.123, compared to 0.077 for TxtLDA. In word similarity tasks, the improvement was even more pronounced, with MixLDA reaching 0.318 versus 0.247 for text alone.
As shown in the results, the choice of similarity measure is critical. Symmetric measures like Jensen-Shannon (JS) divergence proved effective for general similarity, while asymmetric measures like conditional probability $P(w_2|w_1)$ were superior for directional word associations.
Discussion and Conclusion
The inclusion of visual data allows for "visual disambiguation," where perceptual features clarify meanings that might be ambiguous in text alone. For example, the model can learn the visual link between "apple" and "red" even if those words rarely co-occur in the training text.
Despite these gains, the model faces limitations. The "bag-of-visual-words" approach lacks spatial awareness, treating all features equally regardless of their position in the image. Future work could involve more sophisticated image segmentation or non-parametric models that automatically determine the optimal number of topics and visual vocabulary size. Nevertheless, this research demonstrates that grounding language in the visual modality is a significant step toward more human-like semantic representations.
Summary updated