Introduction
General-purpose computer vision—identifying objects in a vacuum—is an "AI-complete" problem. While many vision systems use static, "common sense" context (e.g., "cars are usually on roads"), they often struggle with the unique, dynamic context of a specific image. This paper introduces "visual semantics," a framework for extracting specific visual constraints from text captions to guide computer vision systems. By leveraging a specialized lexicon and a rule-based constraint generator, the authors' system, PICTION, can identify and label human faces in newspaper photographs without requiring pre-stored biometric face models.
The PICTION System
The researchers developed a multi-stage pipeline integrated into PICTION. The core components include:
- Augmented Lexical Database: Standard linguistic resources (WordNet, LDOCE, OALD) are combined with a custom "visual hierarchy." This hierarchy maps words to physical properties like size, shape, and "visual-is-a" relationships. For example, the entry for "hat" includes procedural information that its typical location is "above the head" of a human.
- NLP Module:
- Pre-processing: Uses a Hidden Markov Model (HMM) to detect and categorize proper names (e.g., identifying "New York" as a location vs. "Sean Penn" as a person).
- LFG Parser: Performs syntactic and partial semantic analysis.
- Caption-based Constraint Generator (CBCG): A rule-based system that converts the parse tree into four types of visual constraints: Spatial (left-of), Locative (in the corner), Characteristic (gender, hair color), and Contextual (is this an indoor or outdoor scene?).
- Language-Image Interface (LII): This acts as the "manager," taking the text-derived constraints and using them to direct an Image Understanding (IU) module to find specific "blobs" (faces) that satisfy those constraints.
Identification through Context
Unlike traditional vision systems that have a fixed idea of what a "living room" looks like, PICTION builds a unique model for every image based on its specific caption. It doesn't "recognize" faces in the biometric sense (matching pixels to a database of known people). Instead, it "identifies" them by process of elimination using spatial logic. For instance, if a caption mentions "Sean Penn" and "Toukie Smith," the system finds all face-like blobs and uses spatial constraints (like "left-of") to assign labels.
Results and Insights
The system was evaluated on a database of 50 digitized newspaper photographs and captions from The Buffalo News. PICTION successfully and uniquely identified faces in 62% of the test cases, with partial success in an additional 11%.
Key insights from the study include:
- Visual Hierarchies: Linguistic ontologies like WordNet are insufficient for vision. The authors added "visual-part-of" links (e.g., a "tree" visually consists of a "treetop" and a "trunk") to bridge the gap.
- Collateral Information: Text provides "hints" that drastically reduce the search space for vision algorithms, making a nearly impossible vision task computationally feasible.
- Robustness: Even when low-level vision filters (like a male/female classifier) were ambiguous, spatial constraints (like "adjacent" or "left-of") were often enough to correctly label individuals.
Limitations and Future Work
While promising, the system faces several hurdles. The HMM for name detection, while 90% accurate, still struggles with unknown proper nouns. Furthermore, the system is dependent on "blob theory"; if the initial face locator fails to find a face-like region, the semantic constraints have nothing to bind to. Future work aims to extend these visual hierarchies to more diverse object classes and environments beyond news photographs.
Summary updated