Visual Information in Semantic Representation

Yansong Feng, Mirella LapataNAACL 2010 · OpenReview

Abstract

The question of how meaning might be acquired by young children and represented by adult speakers of a language is one of the most debated topics in cognitive science. Existing semantic representation models are primarily amodal based on information provided by the linguistic input despite ample evidence indicating that the cognitive system is also sensitive to perceptual information. In this work we exploit the vast resource of images and associated documents available on the web and develop a model of multimodal meaning representation which is based on the linguistic and visual context. Experimental results show that a closer correspondence to human data can be obtained by taking the visual modality into account.

Introduction

Traditional computational models of semantics, such as Latent Semantic Analysis (LSA), typically rely on the distributional hypothesis: the idea that words appearing in similar linguistic contexts share similar meanings. However, these "amodal" models ignore a crucial aspect of human cognition—our language is grounded in a physical, visual world. Humans do not learn language from text alone; we learn through a rich stream of perceptual evidence. This paper addresses the gap in cognitive realism by proposing a model that integrates discrete linguistic data with high-dimensional visual information.

Methodology: Multimodal Topic Modeling

The core of this research is the development of a multimodal Latent Dirichlet Allocation (MixLDA) model. To bridge the gap between text and images, the authors treat visual features as a discrete vocabulary of "visual words" (visiterms).

Image Quantization and Representation

Images are converted into a format compatible with text-based topic models through quantization. The researchers explored several methods for extracting visual features, including Scale Invariant Feature Transform (SIFT) to detect local interest points, grid-based sampling, and image segmentation.

Visual feature extraction techniques: (a) segmentation, (b) grid-based, and (c) SIFT interest points.

Once features are extracted, they are clustered using K-means to create a visual vocabulary. Each document in the training corpus is then represented as a "Bag-of-Mixed-Words," containing both standard textual tokens and these newly defined visual words.

The MixLDA Model

The LDA framework is applied to this mixed collection. In MixLDA, a latent "topic" represents a probability distribution over both the textual and visual vocabularies. This allows the model to capture cross-modal associations, such as the visual texture of grass being associated with the word "sports" or "football."

Experimental Results

The model was trained on a corpus of 3,361 documents from the BBC News website, where each article was paired with its original image. Performance was evaluated against human gold standards: the Nelson et al. (1999) word association norms and the WordSim353 similarity dataset.

Performance Comparison

The multimodal approach (MixLDA) consistently outperformed the text-only baseline (TxtLDA). For word association tasks, MixLDA achieved a Pearson’s correlation of 0.123, compared to 0.077 for TxtLDA. In word similarity tasks, the improvement was even more pronounced, with MixLDA reaching 0.318 versus 0.247 for text alone.

Pearson's r correlation for different similarity measures across varying image regions and topics.

As shown in the results, the choice of similarity measure is critical. Symmetric measures like Jensen-Shannon (JS) divergence proved effective for general similarity, while asymmetric measures like conditional probability $P(w_2|w_1)$ were superior for directional word associations.

Discussion and Conclusion

The inclusion of visual data allows for "visual disambiguation," where perceptual features clarify meanings that might be ambiguous in text alone. For example, the model can learn the visual link between "apple" and "red" even if those words rarely co-occur in the training text.

Despite these gains, the model faces limitations. The "bag-of-visual-words" approach lacks spatial awareness, treating all features equally regardless of their position in the image. Future work could involve more sophisticated image segmentation or non-parametric models that automatically determine the optimal number of topics and visual vocabulary size. Nevertheless, this research demonstrates that grounding language in the visual modality is a significant step toward more human-like semantic representations.

Summary updated

Abstract

The question of how meaning might be acquired by young children and represented by adult speakers of a language is one of the most debated topics in cognitive science. Existing semantic representation models are primarily amodal based on information provided by the linguistic input despite ample evidence indicating that the cognitive system is also sensitive to perceptual information. In this work we exploit the vast resource of images and associated documents available on the web and develop a model of multimodal meaning representation which is based on the linguistic and visual context. Experimental results show that a closer correspondence to human data can be obtained by taking the visual modality into account.
BibTeX
@inproceedings{feng2010visual,
  author = {Yansong Feng and Mirella Lapata},
  title = {{Visual Information in Semantic Representation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=BJE_ifZO-H},
}
Figure