A self-organizing multiple-view representation of 3D objects

NeurIPS 1989

TL;DR

This paper introduces a novel two-layer neural network, CLF, that learns to represent 3D objects from multiple 2D views without explicit supervision. The network's ability to generalize to novel viewpoints and its behavior mimicking human psychophysical experiments suggest a new perspective on how the brain might achieve object recognition without explicit "mental rotation" mechanisms.

What Problem Does This Paper Address?

Recognizing 3D objects from varying viewpoints is a fundamental challenge in computer vision and cognitive science. Traditional approaches often rely on either complex 3D object-centered representations or storing a vast number of specific 2D views. Human visual systems, however, seem to exhibit a remarkable ability to recognize objects from novel angles, a phenomenon often attributed to "mental rotation." This paper questions whether this mental rotation is a direct process or an emergent property of a more memory-intensive, view-based representation system, similar to what is observed in human learning.

Methods

The core of the proposed solution is the CLF (conjunctions of localized features) network, a two-layer architecture:

  • F-layer (Input/Feature Layer): A feature map where input features (e.g., vertices of wire-frame objects) are represented.
  • R-layer (Representation Layer): A layer of units that learn to respond to conjunctions of features from the F-layer.

Key mechanisms include:

  • Unsupervised Hebbian Relaxation: Used for learning. When a view is presented, active F-units strengthen their connections to active R-units.
  • Winner-Take-All (WTA): A mechanism to select a few most active R-units to represent a given view.
  • Lateral Connections (L-connections): These connections between R-units are strengthened over time for views of the same object, forming associations. This temporal association is crucial for grouping different views of an object.
  • Threshold Adjustment: R-units that win a WTA competition have their thresholds raised, making them more selective for specific input patterns.

The network learns to associate different views of the same object through temporal proximity, creating "footprints" in the R-layer. This distributed representation, built from associations between view-specific representations, is proposed as an alternative to explicit mental rotation.

Datasets

The paper uses simulated experiments with ten novel 3D wire-frame objects. The training data consists of projections of these objects' vertices from 16 evenly spaced viewpoints. For testing, novel views were generated by rotating objects up to 23 degrees away from learned views.

Key Insights and Ideas

  • Emergent Generalization: The network demonstrates significant generalization to novel views, even without an explicit "rotation" mechanism. This suggests that mental rotation might be an emergent property of learning multiple views.
  • Temporal Association as Grouping: The paper highlights the power of temporal association (Hebbian learning between R-units representing successive views) in grouping different views of the same object, forming a cohesive representation.
  • Practice Alters Representation: Similar to human learning, practice (repeated exposure to views) leads to a change in the network's representation. This change appears to reduce the dependence on orientation, mirroring the shift from mental rotation to direct comparison in human recognition.
  • Footprints as Substrate for Phenomena: The learned "footprints" of objects in the representation layer are proposed as a potential substrate for phenomena like canonical views and mental rotation.

Key Results

  • The CLF network learned to recognize ten 3D objects from different viewpoints.
  • When tested on novel views, the network exhibited substantial generalization capability. For rotations up to 4 degrees, classification was near-perfect, decreasing to 30% at 23 degrees (chance level was 10%).
  • The network's behavior in simulated psychophysical experiments (e.g., dependence of recognition on view orientation) was qualitatively similar to that of human subjects.
  • With practice, the network's performance showed a reduced dependence on orientation, mirroring human adaptation.

Limitations and Future Work

  • Wire-frame Objects: The current model is tested on wire-frame objects, which are simpler than real-world objects with complex textures and shading.
  • Simulated Data: The experiments are simulated, and testing on real-world image data would be a crucial next step.
  • Computational Equivalence: The paper notes that blurring the input before applying it to the F-layer can significantly extend generalization ability, linking the approach to interpolation with Radial Basis Functions. This suggests avenues for further exploration in feature extraction and representation.
  • Further Psychophysical Validation: While the model mimics human behavior, further detailed psychophysical experiments and comparisons are warranted.

Summary updated

BibTeX
@inproceedings{1989self,
  title = {{A self-organizing multiple-view representation of 3D objects}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1989},
}
Figure