Representation and recognition of the spatial organization of three-dimensional shapes

D. Marr, H. K. NishiharaProceedings of the Royal Society of London. Series B. Biological Sciences 1978 · DOI

In this seminal 1977 paper from the MIT Artificial Intelligence Laboratory, David Marr and H.K. Nishihara lay the theoretical groundwork for how biological and artificial systems might represent three-dimensional shapes. They argue that for a representation to be truly useful for recognition, it must be object-centered, modular, and based on volumetric primitives—leading to their famous "3-D Model" representation.

TL;DR

Marr and Nishihara propose a hierarchical, object-centered framework for 3D shape representation using generalized cylinders as primitives. By organizing shape information into a modular hierarchy of "stick figures," the system achieves viewpoint-independent recognition and allows for a conservative, general-to-specific analysis of visual data.

What Problem Does This Paper Address?

The core challenge in computer vision is that retinal (or camera) images are two-dimensional, while the world is three-dimensional. Traditional "viewer-centered" approaches require storing every possible orientation of an object as a separate description, which is computationally expensive and fragile.

The authors seek a representation that satisfies three criteria:

  1. Accessibility: Can it be computed easily from an image?
  2. Scope and Uniqueness: Does it cover a wide range of shapes, and does one shape result in one unique description?
  3. Stability and Sensitivity: Is it robust enough to ignore minor variations (stability) while precise enough to distinguish a horse from a cow (sensitivity)?

Methods

The authors introduce the 3-D Model representation, which is built on three design pillars:

  • Object-Centered Coordinate System: Instead of describing where a limb is relative to the viewer, the model describes it relative to the object's own "natural axes" (e.g., the torso).
  • Volumetric Primitives: They use "generalized cones" (essentially cylinders that can vary in diameter along an axis) to represent parts. This is often visualized as a "stick figure" representation.
  • Modular Organization: Shape information is stored in a hierarchy. A "human" model might consist of a single cylinder at the highest level, which decomposes into a torso and limbs at the next level, which further decomposes into forearms and hands.
Figure 3: The modular hierarchy of a human shape, showing how a complex form is decomposed into simpler volumetric primitives.

To relate the 2D image to these 3D models, the authors describe a Constraint Analysis (a relaxation process). This process uses the projected angles of limbs in a 2D image to "back-calculate" the most likely 3D orientation of the object's axes.

Datasets

Not discussed in the paper. As a foundational theoretical paper from 1977, it does not utilize modern large-scale benchmarks. Instead, the authors demonstrate their theories using illustrative examples of animal silhouettes (donkeys, giraffes) and common objects (buckets).

Key Insights and Ideas

  • The Power of Stick Figures: The authors demonstrate that a small number of sticks can effectively portray animal shapes because these sticks correspond to the "natural axes" of the shape's components.
Figure 1: Pipe-cleaner figures illustrating how animal shapes can be effectively represented by the arrangement of a few natural axes.
  • Modularity for Stability: By decoupling local relationships (like fingers to a hand) from global ones (like a hand to a torso), the representation remains stable. You don't need to re-calculate the entire body just because a finger moves.
  • Conservative Recognition: Recognition should proceed from the general to the specific. A system should first identify an object as a "quadruped" before trying to decide if it is a "biped" or a "bird."
  • Adjunct Relations: The paper introduces a formal way to describe how one axis relates to another using cylindrical coordinates (position, distance, and angle), allowing for a precise but flexible spatial grammar.

Key Results

  • Contour Decomposition: The authors show that the occluding contours (outlines) of an object can be used to reliably find its natural axes, provided the axes aren't too foreshortened.
Figure 6: A step-by-step decomposition of a toy donkey's contour into its component axes.
  • Explaining "Unconventional Views": The theory provides a computational explanation for why patients with certain brain lesions fail to recognize objects from "unconventional" angles (like looking straight down into a bucket). In these views, the "principal axis" is foreshortened, making it impossible to establish the object-centered coordinate system needed for recognition.
Figure 7: A water bucket seen from the side vs. the top. The top-down view obscures the principal axis, making recognition significantly harder.

Limitations and Future Work

  • Scope: The current model is best suited for elongated shapes (generalized cones). It struggles with shapes that lack clear natural axes, such as a crumpled piece of paper or a perfect sphere.
  • Foreshortening: If an object is viewed head-on such that its main axis is hidden, the system fails to initialize the coordinate system. The authors suggest using "parent indexes" (recognizing a component like a face to infer the rest of the body) as a workaround.
  • Derivation from Images: While the paper outlines the representation, the authors acknowledge that more work is needed on the derivation—specifically how to extract these axes from shading, texture, and motion rather than just contours.

Summary updated

Abstract

Abstract The human visual process can be studied by examining the computational problems associated with deriving useful information from retinal images. In this paper, we apply this approach to the problem of representing three-dimensional shapes for the purpose of recognition. 1. Three criteria, accessibility, scope and uniqueness, and stability and sensitivity, are presented for judging the usefulness of a representation for shape recognition. 2. Three aspects of a representation’s design are considered, (i) the representation’s coordinate system, (ii) its primitives, which are the primary units of shape information used in the representation, and (iii) the organization the representation imposes on the information in its descriptions. 3. In terms of these design issues and the criteria presented, a shape representation for recognition should: (i) use an object-centred coordinate system, (ii) include volumetric primitives of varied sizes, and (iii) have a modular organization. A representation based on a shape’s natural axes (for example the axes identified by a stick figure) follows directly from these choices. 4. The basic process for deriving a shape description in this representation must involve: (i) a means for identifying the natural axes of a shape in its image and (ii) a mechanism for transforming viewer-centred axis specifications to specifications in an object-centred coordinate system. 5. Shape recognition involves: (i) a collection of stored shape descriptions, and (ii) various indexes into the collection that allow a newly derived description to be associated with an appropriate stored description. The most important of these indexes allows shape recognition to proceed conservatively from the general to the specific based on the specificity of the information available from the image. 6. New constraints supplied by a conservative recognition process can be used to extract more information from the image. A relaxation process for carrying out this constraint analysis is described.
BibTeX
@inproceedings{marr1978representation,
  author = {D. Marr and H. K. Nishihara},
  title = {{Representation and recognition of the spatial organization of three-dimensional shapes}},
  booktitle = {Proceedings of the Royal Society of London. Series B. Biological Sciences},
  year = {1978},
  doi = {10.1098/rspb.1978.0020},
}
Figure