In this seminal 1977 paper from the MIT Artificial Intelligence Laboratory, David Marr and H.K. Nishihara lay the theoretical groundwork for how biological and artificial systems might represent three-dimensional shapes. They argue that for a representation to be truly useful for recognition, it must be object-centered, modular, and based on volumetric primitives—leading to their famous "3-D Model" representation.
TL;DR
Marr and Nishihara propose a hierarchical, object-centered framework for 3D shape representation using generalized cylinders as primitives. By organizing shape information into a modular hierarchy of "stick figures," the system achieves viewpoint-independent recognition and allows for a conservative, general-to-specific analysis of visual data.
What Problem Does This Paper Address?
The core challenge in computer vision is that retinal (or camera) images are two-dimensional, while the world is three-dimensional. Traditional "viewer-centered" approaches require storing every possible orientation of an object as a separate description, which is computationally expensive and fragile.
The authors seek a representation that satisfies three criteria:
- Accessibility: Can it be computed easily from an image?
- Scope and Uniqueness: Does it cover a wide range of shapes, and does one shape result in one unique description?
- Stability and Sensitivity: Is it robust enough to ignore minor variations (stability) while precise enough to distinguish a horse from a cow (sensitivity)?
Methods
The authors introduce the 3-D Model representation, which is built on three design pillars:
- Object-Centered Coordinate System: Instead of describing where a limb is relative to the viewer, the model describes it relative to the object's own "natural axes" (e.g., the torso).
- Volumetric Primitives: They use "generalized cones" (essentially cylinders that can vary in diameter along an axis) to represent parts. This is often visualized as a "stick figure" representation.
- Modular Organization: Shape information is stored in a hierarchy. A "human" model might consist of a single cylinder at the highest level, which decomposes into a torso and limbs at the next level, which further decomposes into forearms and hands.
To relate the 2D image to these 3D models, the authors describe a Constraint Analysis (a relaxation process). This process uses the projected angles of limbs in a 2D image to "back-calculate" the most likely 3D orientation of the object's axes.
Datasets
Not discussed in the paper. As a foundational theoretical paper from 1977, it does not utilize modern large-scale benchmarks. Instead, the authors demonstrate their theories using illustrative examples of animal silhouettes (donkeys, giraffes) and common objects (buckets).
Key Insights and Ideas
- The Power of Stick Figures: The authors demonstrate that a small number of sticks can effectively portray animal shapes because these sticks correspond to the "natural axes" of the shape's components.
- Modularity for Stability: By decoupling local relationships (like fingers to a hand) from global ones (like a hand to a torso), the representation remains stable. You don't need to re-calculate the entire body just because a finger moves.
- Conservative Recognition: Recognition should proceed from the general to the specific. A system should first identify an object as a "quadruped" before trying to decide if it is a "biped" or a "bird."
- Adjunct Relations: The paper introduces a formal way to describe how one axis relates to another using cylindrical coordinates (position, distance, and angle), allowing for a precise but flexible spatial grammar.
Key Results
- Contour Decomposition: The authors show that the occluding contours (outlines) of an object can be used to reliably find its natural axes, provided the axes aren't too foreshortened.
- Explaining "Unconventional Views": The theory provides a computational explanation for why patients with certain brain lesions fail to recognize objects from "unconventional" angles (like looking straight down into a bucket). In these views, the "principal axis" is foreshortened, making it impossible to establish the object-centered coordinate system needed for recognition.
Limitations and Future Work
- Scope: The current model is best suited for elongated shapes (generalized cones). It struggles with shapes that lack clear natural axes, such as a crumpled piece of paper or a perfect sphere.
- Foreshortening: If an object is viewed head-on such that its main axis is hidden, the system fails to initialize the coordinate system. The authors suggest using "parent indexes" (recognizing a component like a face to infer the rest of the body) as a workaround.
- Derivation from Images: While the paper outlines the representation, the authors acknowledge that more work is needed on the derivation—specifically how to extract these axes from shading, texture, and motion rather than just contours.
Summary updated