Abstract
Topics generated automatically, e.g. using LDA, are now widely used in Computational Linguistics. Topics are normally represented as a set of keywords, often the n terms in a topic with the highest marginal probabilities. We introduce an alternative approach in which topics are represented using images. Candidate images for each topic are retrieved from the web by querying a search engine using the top n terms. The most suitable image is selected from this set using a graph-based algorithm which makes use of textual information from the metadata associated with each image and features extracted from the images themselves. We show that the proposed approach significantly outperforms several baselines and can provide images that are useful to represent a topic.
Introduction
In Natural Language Processing (NLP), topic modeling is a standard technique for summarizing large document collections. However, the traditional output—a list of top $n$ terms (e.g., wine, bottle, grape, flavor)—can be mentally taxing for humans to interpret, especially when dealing with dozens of topics.
This research explores a multimodal alternative: representing topics using images. Images are language-independent and can be processed by the human brain much faster than text. The primary challenge lies in selection: querying a search engine with topic keywords often returns noisy or irrelevant results. This paper introduces a novel graph-based algorithm to automatically identify the single most representative image for a latent topic.
Methodology
The proposed pipeline transforms a set of keywords into a single representative image through three main stages:
- Candidate Retrieval: The top five terms of a topic are used to query Google Image Search (restricted to Wikipedia/Creative Commons), collecting the top 20 candidate images.
- Feature Extraction:
- Textual Metadata: Page titles and filenames associated with each image are extracted.
- Visual Features: Scale-Invariant Feature Transform (SIFT) descriptors are used to identify key points in images. These are clustered into a "vocabulary" of 1,000 visual words to create a Bag-of-Visual-Words (BOVW) representation.
- Graph-Based Ranking: A graph is constructed where each node represents a candidate image. Edge weights are determined by similarity—either textual (metadata) or visual (BOVW). The system then applies ranking algorithms like PageRank and Personalised PageRank (PPR). In PPR, the algorithm is biased toward images whose metadata shows high Pointwise Mutual Information (PMI) with the original topic keywords.
Evaluation and Results
The approach was evaluated on 300 topics: 100 from New York Times articles and 200 from Wikipedia. A "gold standard" dataset was created via crowdsourcing, where over 270 participants rated 6,000 topic-image pairs on a scale of 0 (unsuitable) to 3 (very suitable).
Performance Metrics
The system's performance was measured using the Top-1 Average Rating (the score of the system's top pick) and nDCG.
- The Winner: The Personalised PageRank model combining visual features and PMI ($PR_{vis} + Per(PMI)$) achieved a Top-1 rating of 2.00.
- Baselines: This significantly outperformed Google Image Search's default ranking (1.89), a Word Overlap baseline (1.85), and random selection (1.79).
- Human Performance: In a separate study, humans achieved an average rating of 2.24, indicating that the automated system is approaching human-level selection capability.
Discussion and Insights
The study revealed that multimodal approaches are superior. Specifically, using visual similarity to weight graph edges outperformed text-based similarity, suggesting that visual consistency among candidates is a strong indicator of relevance. Furthermore, simple word association (PMI) proved more robust than structured knowledge-based measures like Explicit Semantic Analysis (ESA) for noisy metadata.
However, limitations remain. Abstract topics (e.g., "Algebraic Number Theory") are inherently difficult to visualize. Additionally, polysemy can lead to errors, such as a search for "Plants" returning car images due to shared taxonomic terms.
Conclusion
By leveraging both visual features and textual metadata within a graph-based ranking framework, this research demonstrates that images can effectively represent latent topics. This multimodal approach offers a more intuitive way for users to navigate and understand large-scale text corpora.
Summary updated