Unsupervised Learning of Visual Sense Models for Polysemous Words

Kate Saenko, Trevor DarrellNeurIPS 2008 · OpenReview

Introduction

This paper introduces a fully unsupervised method to train object classifiers from noisy web search results by disambiguating word meanings—a challenge known as polysemy. By combining sparse dictionary definitions with Latent Dirichlet Allocation (LDA) on web text, the authors demonstrate how to automatically filter and re-rank images to train robust visual models without any manual labeling.

The Challenge of Polysemy

When computer vision researchers want to train a new object model, they often turn to image search engines to gather training data. However, language is inherently messy. If you search for "bass," you retrieve a mix of fish and guitars; search for "mouse," and you get rodents, computer peripherals, and cartoon characters.

Examples of image search results for the polysemous word "mouse", showing rodents, computer peripherals, and cartoon characters.

Existing unsupervised methods for learning from the web often struggle with this ambiguity. Some assume the majority of search results belong to the desired category (which fails for highly ambiguous words), while others require a small "seed" set of manually labeled images to get started. The core challenge addressed in this work is: How can we build a specific visual model using only a raw keyword search and a dictionary, without a human ever labeling a single image?

A Dictionary-Driven Methodology

The authors propose a multimodal approach that leverages surrounding text to "clean" the visual data. The process follows three main steps:

  1. Discovering a Latent Text Space: Because dictionary definitions are very short, they lack the statistical power needed to match against diverse web text. The authors first run Latent Dirichlet Allocation (LDA) on a large corpus of unlabeled web pages related to the keyword. This discovers latent "topics" (clusters of co-occurring words) that represent different ways the word is used online.
  2. Mapping Dictionary Senses to Topics: The system extracts the formal definition of a word sense from WordNet, including synonyms and hypernyms (e.g., "electronic device" for a computer mouse). It then calculates the likelihood of each dictionary sense belonging to the latent LDA topics. This effectively bridges the gap between formal linguistics and informal web text.
  3. Image Re-ranking and Visual Training: For every image returned by a search engine, the algorithm analyzes the "bag-of-words" in the surrounding HTML text. Using the model from Step 2, it assigns a probability that the image belongs to a specific dictionary sense. The top-ranked images for a sense are then used as positive training data for a Support Vector Machine (SVM) classifier, utilizing SIFT descriptors and a "visual words" approach for image features.

Experimental Setup

The method was evaluated on a dataset of five highly polysemous words: bass, face, mouse, speaker, and watch.

  • Keyword Dataset: Images and surrounding HTML text pulled from Yahoo Image Search.
  • Text-only Dataset: Approximately 1,000 web pages per keyword used to train the LDA model.
  • Sense-term Dataset: Images found by searching for more specific terms (e.g., "percoid bass"), used as a baseline comparison.

While the training process was entirely unsupervised, a human annotator labeled the images (as "good," "partial," or "unrelated") strictly for evaluation purposes.

Key Findings

  • Dictionaries as a "Cold Start" Solution: The most elegant insight is using a dictionary as the sole source of supervision. It replaces the need for human-labeled images by providing a semantic anchor.
  • LDA as a Semantic Bridge: Dictionary definitions are too sparse to match messy web pages directly. Using LDA to create a latent space allows the system to realize that words like "fishing" and "lure" are related to the "fish" sense of "bass," even if they aren't in the formal definition.
  • Improved Retrieval and Classification: The dictionary-based model significantly outperformed the original search engine rankings. Furthermore, SVMs trained on the dictionary-ranked images were more accurate at identifying the correct sense in unseen images than those trained on baseline search-term images. The system even successfully distinguished between subtle senses, such as "loudspeaker" versus "human speaker."

Limitations and Future Work

The method relies on the assumption that different dictionary senses are visually distinct; if two senses look identical but have different meanings, a purely visual classifier will still struggle. Additionally, the re-ranking mechanism fails if an image is embedded in a page with generic or missing text. Future work could involve automatically determining which dictionary senses are likely to have a coherent visual representation before attempting to train a model.

Summary updated

Abstract

Polysemy is a problem for methods that exploit image search engines to build object category models. Existing unsupervised approaches do not take word sense into consideration. We propose a new method that uses a dictionary to learn models of visual word sense from a large collection of unlabeled web data. The use of LDA to discover a latent sense space makes the model robust despite the very limited nature of dictionary definitions. The definitions are used to learn a distribution in the latent space that best represents a sense. The algorithm then uses the text surrounding image links to retrieve images with high probability of a particular dictionary sense. An object classifier is trained on the resulting sense-specific images. We evaluate our method on a dataset obtained by searching the web for polysemous words. Category classification experiments show that our dictionary-based approach outperforms baseline methods.
BibTeX
@inproceedings{saenko2008unsupervised,
  author = {Kate Saenko and Trevor Darrell},
  title = {{Unsupervised Learning of Visual Sense Models for Polysemous Words}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=BkVTWw-OWH},
}
Figure