A Principle for Unsupervised Hierarchical Decomposition of Visual Scenes

Michael Mozer

How does a visual system—biological or artificial—decide where one object ends and another begins, or which "bits" of a scene constitute a meaningful part?

Historically, computer vision models relied heavily on pre-defined object models or supervised labels (e.g., a "teacher" explicitly pointing out a "nose"). However, humans naturally learn to decompose scenes into complex hierarchies—from the overall scene down to objects, parts, and subparts—without explicit instruction.

Hierarchical structure of a visual scene

In this paper, the author seeks a fundamental, unsupervised principle that explains how these hierarchical relationships emerge purely from the raw statistics of the visual environment. The challenge is twofold: defining a mathematical principle of "part-hood" and finding a way for connectionist (neural) networks to represent hierarchical trees without relying on symbolic pointers.

The Regularity Principle and the MAGIC Model

To solve this, the author introduces the Regularity Principle. This hypothesis states that the internal structure of a "part" is highly regular and consistent across different instances, while the spatial relationship between different parts is more variable. For example, the distance between a person's eyes and nose is relatively fixed, but the distance between their hand and a tool they are holding varies wildly.

To implement this principle in a neural network without using rigid symbolic trees, the author proposes a Flat Representation via Angular Tags. Each feature is assigned a "tag" (a value from $0$ to $2\pi$). Features belonging to the same part are driven to have similar angular values. This approach is inspired by the neurophysiological theory of binding by synchrony, where neurons representing the same object fire in phase.

These concepts are operationalized in the MAGIC Model, a two-layer recurrent neural network acting as a bottleneck autoencoder:

  • Input: Spatiotopic arrays of oriented line segments.
  • Mechanism: Hidden units detect local configurations of features. If a configuration is "regular" (frequently seen during training), the network forces the tags of those features to align. If the relationship is irregular, weak inhibitory connections push the tags apart.
  • Dynamics: The system uses a mean-field energy functional to settle into a stable state of tag assignments through a relaxation process.

Key Insights and Experimental Results

The model was tested on synthetic, controlled datasets to evaluate specific psychological and structural hypotheses.

  • Statistical Part-hood: The most profound insight is that "objectness" is a statistical property. The system doesn't need to know what a "head" is to identify it; it just needs to notice that certain features consistently appear together. In tests with simple line pairs, MAGIC successfully grouped segments into distinct lines because the junction between the lines was inconsistent across training, prompting the model to "segment" the scene.
  • Hierarchy as Phase: By using a continuous angular spectrum, the model can represent sub-parts. If a "head" has a tag range of $0.1$ to $0.5$, the "eyes" might cluster tightly at $0.15$ and the "nose" at $0.45$.
  • Replicating Human Psychology: The model was tested on 2D contour versions of "Venusian rocks" (deformed spheres with specific, consistent protrusions) used in human psychophysical studies. MAGIC's tags clustered significantly more tightly for "target" parts (variance of $0.30$) than for alternative, irregular parts (variance of $0.68$). This perfectly matches human data showing that consistent protrusions become salient parts.
  • Prior Learning: The model accounted for the "ontogeny" of parts. If pre-trained on Part A and Part B separately, it was much better at decomposing a complex object containing both than a "naive" model trained only on the combined object.

Limitations and Future Work

While groundbreaking, the model has limitations. It relies on primitive features (simple oriented line segments), making it too inflexible for certain spatial reasoning tasks.

The box and lid problem

Specifically, the author notes the model cannot currently handle the "box and lid" problem (illustrated above). Because it lacks a representation of relative spatial relationships and only sees absolute orientation on the image plane, it struggles when the internal configuration of a part (like a hinged lid) changes its absolute orientation.

Furthermore, the current system lacks scale and rotation invariance. If an object rotates, its "regularity" in terms of fixed line orientations is lost. Finally, while the tags provide a "flat" representation, the paper leaves open the "read-out" problem: how a higher-level system would interpret these continuous tags to make discrete decisions without an external clustering algorithm. The author suggests that selecting specific tag ranges for processing may be deeply related to visual attention, paving the way for future research.

Summary updated

Abstract

Structure in a visual scene can be described at many levels of granular(cid:173) ity. At a coarse level, the scene is composed of objects; at a finer level, each object is made up of parts, and the parts of subparts. In this work, I propose a simple principle by which such hierarchical structure can be extracted from visual scenes: Regularity in the relations among different parts of an object is weaker than in the internal structure of a part. This principle can be applied recursively to define part-whole relationships among elements in a scene. The principle does not make use of object models, categories, or other sorts of higher-level knowledge; rather, part-whole relationships can be established based on the statistics of a set of sample visual scenes. I illustrate with a model that performs unsu(cid:173) pervised decomposition of simple scenes. The model can account for the results from a human learning experiment on the ontogeny of part(cid:173) whole relationships.
BibTeX
@inproceedings{mozer1998principle,
  author = {Michael Mozer},
  title = {{A Principle for Unsupervised Hierarchical Decomposition of Visual Scenes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=S1NDkKWObr},
}

Loading related content...

Figure