TL;DR
This paper introduces "body plans," a hierarchical representation for recognizing humans and animals in complex, unconstrained images. By combining low-level color and texture cues with high-level geometric constraints—modeled as assemblies of nearly cylindrical parts—the authors demonstrate a robust method for semantic content-based image retrieval.
What Problem Does This Paper Address?
In 1997, the "semantic gap" in computer vision was a chasm. While internet usage was exploding, content-based retrieval systems were largely limited to matching global color histograms or simple textures. They couldn't answer abstract queries like "find people" or "find horses" in cluttered, real-world scenes.
The authors argue that traditional segmentation—which groups pixels by coherent color or texture—is insufficient for object recognition. Instead, they propose that segmentation must be "model-driven": you can only find an object if you already have a structural plan for what that object looks like.
Methods
The core of the approach is the body plan, a sequence of grouping stages that mirrors the physical layout of a body.
- Low-level Filtering: The system first identifies regions that could be skin or hide using color and texture filters.
- Primitive Extraction: Within these regions, it looks for "straight ribbons"—image segments with near-parallel sides that represent the 2D projections of cylinders (limbs, torsos). This is done using Canny edge detection and Hough transforms.
- Hierarchical Grouping: These ribbons are assembled into a hierarchy. For a horse, the system first finds legs, necks, and bodies, then pairs them (body-leg, body-neck), then triples, and finally full assemblies.
- Statistical Learning: Rather than hard-coding every rule, the authors use statistical learning to infer the decision boundaries for these assemblies. They use a technique involving "augmented feature vectors" to train the entire hierarchy simultaneously, ensuring that early-stage classifiers pass along segments that are likely to form valid full bodies later.
- Geometric Invariance: To handle different poses and viewpoints, segments are measured in a "canonical frame" where a distinguished segment (like the torso) is centered and aligned.
Datasets
The authors evaluated their approach on two distinct tasks:
- Naked Humans: A target set of 565 images (collected from the internet and scanned from books) and a control set of 4,302 assorted images.
- Horses: A target set of 100 images from the Corel "Arabian horses" library and 1,086 unrelated control images.
Key Insights and Ideas
- Kinematic Constraints as Filters: The most powerful idea here is that the physical geometry of skeletons and ligaments limits how parts can appear together. These constraints are "non-trivial," meaning they can effectively filter out accidental alignments of background clutter.
- The Response Ratio: The authors emphasize the "response ratio" (the density of test images in the output vs. the input). They show that while a simple skin filter has high recall, it has a poor response ratio. Adding geometric grouping stages dramatically increases selectivity.
- Complexity as a Guide: As the geometric complexity of a group increases (e.g., moving from a pair to a quadruple of segments), the likelihood that it represents a real object increases significantly. Complex geometric groups are "extremely distinctive."
Key Results
The horse recognition system achieved a recall of 15% with a response ratio of 23. While 15% recall might seem low by modern standards, a response ratio of 23 means the system is exceptionally good at filtering out "noise" from the control set.
As shown in Figure 2 and Figure 3, the "grouper" (the geometric body plan) consistently outperformed simple color/texture filters. In the horse query, the system successfully recovered horses in a wide range of aspects and poses, as seen in Figure 4.
Limitations and Future Work
- Impoverished Primitives: The current model relies on simple ribbons and ignores shading or internal texture details (like the specific patterns of a leopard's pelt).
- Viewpoint Gaps: The system struggles with specific viewpoints, such as head-on or overhead views, where the standard "ribbon" assembly doesn't match the 2D projection. The authors suggest adding more specific classifiers to the body plan to handle these cases.
- Implementation Bottlenecks: The process of finding narrow ribbons (like legs) can sometimes generate an overwhelming number of local symmetries, leading to computational inefficiency or errors in cluttered scenes.
- Unified Models: The paper uses separate classifiers for humans and horses. A major future goal is building a single process that can differentiate between various animals while sharing the same underlying geometric primitives.
Summary updated