Visual Semantics: Extracting Visual information from Text Accompanying Pictures

Rohini K. Srihari, Debra T. BurhansAAAI 1994

Introduction

General-purpose computer vision—identifying objects in a vacuum—is an "AI-complete" problem. While many vision systems use static, "common sense" context (e.g., "cars are usually on roads"), they often struggle with the unique, dynamic context of a specific image. This paper introduces "visual semantics," a framework for extracting specific visual constraints from text captions to guide computer vision systems. By leveraging a specialized lexicon and a rule-based constraint generator, the authors' system, PICTION, can identify and label human faces in newspaper photographs without requiring pre-stored biometric face models.

The PICTION System

The researchers developed a multi-stage pipeline integrated into PICTION. The core components include:

  • Augmented Lexical Database: Standard linguistic resources (WordNet, LDOCE, OALD) are combined with a custom "visual hierarchy." This hierarchy maps words to physical properties like size, shape, and "visual-is-a" relationships. For example, the entry for "hat" includes procedural information that its typical location is "above the head" of a human.
  • NLP Module:
    1. Pre-processing: Uses a Hidden Markov Model (HMM) to detect and categorize proper names (e.g., identifying "New York" as a location vs. "Sean Penn" as a person).
  • LFG Parser: Performs syntactic and partial semantic analysis.
  • Caption-based Constraint Generator (CBCG): A rule-based system that converts the parse tree into four types of visual constraints: Spatial (left-of), Locative (in the corner), Characteristic (gender, hair color), and Contextual (is this an indoor or outdoor scene?).
  • Language-Image Interface (LII): This acts as the "manager," taking the text-derived constraints and using them to direct an Image Understanding (IU) module to find specific "blobs" (faces) that satisfy those constraints.

Identification through Context

Unlike traditional vision systems that have a fixed idea of what a "living room" looks like, PICTION builds a unique model for every image based on its specific caption. It doesn't "recognize" faces in the biometric sense (matching pixels to a database of known people). Instead, it "identifies" them by process of elimination using spatial logic. For instance, if a caption mentions "Sean Penn" and "Toukie Smith," the system finds all face-like blobs and uses spatial constraints (like "left-of") to assign labels.

The PICTION system identifying faces based on caption constraints. (a) Original image, (b) candidate face regions, (c) and (d) identified individuals.

Results and Insights

The system was evaluated on a database of 50 digitized newspaper photographs and captions from The Buffalo News. PICTION successfully and uniquely identified faces in 62% of the test cases, with partial success in an additional 11%.

Key insights from the study include:

  • Visual Hierarchies: Linguistic ontologies like WordNet are insufficient for vision. The authors added "visual-part-of" links (e.g., a "tree" visually consists of a "treetop" and a "trunk") to bridge the gap.
  • Collateral Information: Text provides "hints" that drastically reduce the search space for vision algorithms, making a nearly impossible vision task computationally feasible.
  • Robustness: Even when low-level vision filters (like a male/female classifier) were ambiguous, spatial constraints (like "adjacent" or "left-of") were often enough to correctly label individuals.

Limitations and Future Work

While promising, the system faces several hurdles. The HMM for name detection, while 90% accurate, still struggles with unknown proper nouns. Furthermore, the system is dependent on "blob theory"; if the initial face locator fails to find a face-like region, the semantic constraints have nothing to bind to. Future work aims to extend these visual hierarchies to more diverse object classes and environments beyond news photographs.

Summary updated

BibTeX
@inproceedings{srihari1994visual,
  author = {Rohini K. Srihari and Debra T. Burhans},
  title = {{Visual Semantics: Extracting Visual information from Text Accompanying Pictures}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure