Body plans

D.a. Forsyth, M.m. Fleck

TL;DR

This paper introduces "body plans," a hierarchical representation for recognizing humans and animals in complex, unconstrained images. By combining low-level color and texture cues with high-level geometric constraints—modeled as assemblies of nearly cylindrical parts—the authors demonstrate a robust method for semantic content-based image retrieval.

What Problem Does This Paper Address?

In 1997, the "semantic gap" in computer vision was a chasm. While internet usage was exploding, content-based retrieval systems were largely limited to matching global color histograms or simple textures. They couldn't answer abstract queries like "find people" or "find horses" in cluttered, real-world scenes.

The authors argue that traditional segmentation—which groups pixels by coherent color or texture—is insufficient for object recognition. Instead, they propose that segmentation must be "model-driven": you can only find an object if you already have a structural plan for what that object looks like.

Methods

The core of the approach is the body plan, a sequence of grouping stages that mirrors the physical layout of a body.

  1. Low-level Filtering: The system first identifies regions that could be skin or hide using color and texture filters.
  2. Primitive Extraction: Within these regions, it looks for "straight ribbons"—image segments with near-parallel sides that represent the 2D projections of cylinders (limbs, torsos). This is done using Canny edge detection and Hough transforms.
  3. Hierarchical Grouping: These ribbons are assembled into a hierarchy. For a horse, the system first finds legs, necks, and bodies, then pairs them (body-leg, body-neck), then triples, and finally full assemblies.
  4. Statistical Learning: Rather than hard-coding every rule, the authors use statistical learning to infer the decision boundaries for these assemblies. They use a technique involving "augmented feature vectors" to train the entire hierarchy simultaneously, ensuring that early-stage classifiers pass along segments that are likely to form valid full bodies later.
  5. Geometric Invariance: To handle different poses and viewpoints, segments are measured in a "canonical frame" where a distinguished segment (like the torso) is centered and aligned.
Figure 1: The body plan used for horses. Each circle represents a classifier, with an icon indicating the ap- pearance of the assembly. A n arrow indicates that the classifier at the arrowhead uses segments passed by the classifier at the tail. Constraints exist between groups, too; for example, a body-leg-neck classifier will attempt to form traples only out of pairs that share the same body.

Datasets

The authors evaluated their approach on two distinct tasks:

  • Naked Humans: A target set of 565 images (collected from the internet and scanned from books) and a control set of 4,302 assorted images.
  • Horses: A target set of 100 images from the Corel "Arabian horses" library and 1,086 unrelated control images.

Key Insights and Ideas

  • Kinematic Constraints as Filters: The most powerful idea here is that the physical geometry of skeletons and ligaments limits how parts can appear together. These constraints are "non-trivial," meaning they can effectively filter out accidental alignments of background clutter.
  • The Response Ratio: The authors emphasize the "response ratio" (the density of test images in the output vs. the input). They show that while a simple skin filter has high recall, it has a poor response ratio. Adding geometric grouping stages dramatically increases selectivity.
  • Complexity as a Guide: As the geometric complexity of a group increases (e.g., moving from a pair to a quadruple of segments), the likelihood that it represents a real object increases significantly. Complex geometric groups are "extremely distinctive."

Key Results

The horse recognition system achieved a recall of 15% with a response ratio of 23. While 15% recall might seem low by modern standards, a response ratio of 23 means the system is exceptionally good at filtering out "noise" from the control set.

As shown in Figure 2 and Figure 3, the "grouper" (the geometric body plan) consistently outperformed simple color/texture filters. In the horse query, the system successfully recovered horses in a wide range of aspects and poses, as seen in Figure 4.

Figure 4: All images returned from a control set of 1086 and a test set of 100 images, for the horse query with robustness parameter set to the most selective value. The first line of horse images comes from the training set; the rest from the test set. A further four control images could be expected to come from the im- ages that passed the hide filter, but overwhelmed the ribbon finding algorithm. The test images recovered contain horses in a wide range of aspects; one control image contains an animal that might reasonably pass for a horse.

Limitations and Future Work

  • Impoverished Primitives: The current model relies on simple ribbons and ignores shading or internal texture details (like the specific patterns of a leopard's pelt).
  • Viewpoint Gaps: The system struggles with specific viewpoints, such as head-on or overhead views, where the standard "ribbon" assembly doesn't match the 2D projection. The authors suggest adding more specific classifiers to the body plan to handle these cases.
  • Implementation Bottlenecks: The process of finding narrow ribbons (like legs) can sometimes generate an overwhelming number of local symmetries, leading to computational inefficiency or errors in cluttered scenes.
  • Unified Models: The paper uses separate classifiers for humans and horses. A major future goal is building a single process that can differentiate between various animals while sharing the same underlying geometric primitives.

Summary updated

Abstract

This paper describes a representation for people and animals, called a body plan, which is adapted to segmentation and to recognition in complex environments. The representation is an organized collection of grouping hints obtained from a combination of constraints on color and texture and constraints on geometric properties such as the structure of individual parts and the relationships between parts. Body plans can be learned from image data, using established statistical learning techniques. The approach is illustrated with two examples of programs that successfully use body plans for recognition: one example involves determining whether a picture contains a scantily clad human, using a body plan built by hand; the other involves determining whether a picture contains a horse, using a body plan learned from image data. In both cases, the system demonstrates excellent performance on large, uncontrolled test sets and very large and diverse control sets.
BibTeX
@inproceedings{forsythbody,
  author = {D.A. Forsyth and M.M. Fleck},
  title = {{Body plans}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  doi = {10.1109/cvpr.1997.609399},
}
Figure