In the quest to build robots that can learn about the world on the fly, researchers often turn to the vast, messy library of the internet. If you tell a robot to "find a mouse," the easiest way for it to learn what that looks like is to scrape images from a search engine. But there’s a catch: language is messy. A search for "mouse" returns rodents, computer peripherals, and—occasionally—metaphorical references to "timid people."
In their paper, "Filtering Abstract Senses From Image Search Results," Kate Saenko and Trevor Darrell introduce WISDOM (Web Image Sense DisambiguatiOn Model), a system designed to help machines distinguish between the physical objects we want them to see and the abstract concepts that clutter our search results.
The Challenge of Polysemy
The core challenge addressed in this research is polysemy—the fact that one word can have many meanings. When a robot needs to learn a visual model for a "crane," it might be swamped by images of the bird, the construction equipment, or even abstract logos and illustrations related to the surname "Crane."
Existing methods often use image clustering to group similar-looking results, but these clusters are "blind." A cluster might be visually consistent (e.g., a group of abstract logos) without the system knowing if that cluster represents a physical object or a metaphorical concept. The authors argue that for a robot to function in the real world, it must be able to automatically discard these abstract "outlier" senses without human intervention.
The WISDOM Approach
To solve this, the authors propose a multimodal approach that doesn't require any manual labeling. The WISDOM pipeline automatically filters out "abstract" meanings from image search results by combining dictionary definitions with probabilistic models of text and images. It consists of three main pillars:
- The Dictionary Model: The system uses WordNet, a massive semantic database. It looks at the "hypernym tree" (the hierarchy of meanings) for a word. If a sense is a descendant of nodes like "artifact," "animal," or "body part," WISDOM flags it as concrete. If it isn't, it's considered abstract.
- Multimodal Latent Topics: WISDOM uses Latent Dirichlet Allocation (LDA) to discover hidden topics in two domains:
- Text: It analyzes the 100-word window of text surrounding an image link on a webpage.
- Visual: It uses a "bag-of-words" approach for images, extracting SIFT features and clustering them into visual "words."
- Topic Grounding and Adaptation: This is the "secret sauce." The system relates the latent topics found in the search results back to the dictionary definitions. Crucially, it performs an adaptation step: it takes general web-text topics and refines them using the specific text found next to the images, ensuring the model doesn't get confused by "off-topic" web content.
Datasets and Evaluation
The researchers evaluated WISDOM on three distinct datasets to prove its real-world viability:
- MIT-ISD: 5 polysemous words (bass, face, mouse, speaker, watch).
- UIUC-ISD: 3 words (bass, crane, squash).
- OFFICE Dataset: A new collection of 10 common office objects (cellphone, fork, hammer, keyboard, mug, pliers, scissors, stapler, telephone, watch) used to test how well the filtered images could train an actual object classifier.
Key Insights and Results
- Metadata is a Goldmine: The paper highlights that WordNet contains "lexical tags" (like
<artifact>or<animal>) that are incredibly useful for computer vision but were previously underutilized for filtering search results. - Text and Vision are Complementary: Text is excellent at identifying the intent or high-level sense of a page, while visual features are better at grouping specific physical manifestations of an object.
- Unsupervised is Feasible: By grounding statistical models (LDA) in a structured ontology (WordNet), we can achieve "human-like" discernment of abstract vs. concrete concepts without a human ever having to click "this is a picture of a real mug."
Performance Boosts: In almost every category, WISDOM’s Recall-Precision Curves (RPCs) significantly outperformed the raw Yahoo Image Search baseline. It was particularly effective at filtering out logos, sketches, and metaphorical "clutter."
When training a 10-way SVM classifier on office objects, the baseline (using raw search results) achieved 27% accuracy. Models trained on WISDOM-filtered images reached 35% accuracy—a 25% relative improvement. The "mug" category saw a massive boost because WISDOM successfully filtered out non-object senses that typically confuse standard search engines.
Limitations and Future Work
While highly effective, WISDOM has room to grow. It struggled with ambiguous concrete senses, such as "face-13," defined in WordNet as "a vertical surface." While technically an artifact, it is visually too broad to be useful for training a specific object detector.
Additionally, the current model weights text and image features equally. The authors suggest that dynamically weighting these based on the specific word could yield even better results. Finally, while the system excels at separating "abstract" from "concrete," distinguishing between two visually and textually similar concrete senses (e.g., different types of "cranes") remains an open challenge.
Summary updated