Discovering Viewpoint-Invariant Relationships That Characterize Objects

Richard S. Zemel, Geoffrey E. HintonNeurIPS 1990 · OpenReview

TL;DR

This paper introduces an unsupervised learning approach that allows a neural network to "discover" the spatial relationships that define an object. By maximizing the mutual information between predictions made by different parts of the network, the system learns to recognize objects across various positions, orientations, and sizes without ever being told what those parameters are.

What Problem Does This Paper Address?

In computer vision, recognizing an object regardless of its "viewpoint"—its specific translation, rotation, or scale in an image—is a classic challenge. Traditionally, researchers solved this by either manually normalizing the image (e.g., centering the object) or hard-coding invariance into the network architecture (like modern CNNs do with translation).

The authors, Zemel and Hinton, tackle a much harder version of this problem: How can a system learn viewpoint-invariant recognition entirely on its own? They want a model that can look at a pile of unlabelled images of an object in different poses and figure out the underlying spatial constraints that make that object "it," all without a teacher providing labels or viewpoint coordinates.

Methods

The core of the paper is the IMAX learning procedure. The approach is based on the "viewpoint consistency constraint": if two features belong to the same rigid object, they should both be able to predict the same global position, orientation, and size for that object.

Network architecture maximizing mutual information between two halves
  1. Split Architecture: The network is divided into two halves. Each half "sees" a different fragment or feature of the object.
  2. Mutual Information Maximization: Each half produces a 4-parameter output (representing the 4 degrees of freedom in 2D: x, y, orientation, and scale). The training objective is to maximize the mutual information between these two sets of outputs.
  3. The Objective Function: The authors use a specific Gaussian-based formula for mutual information: $$I(a; b) = \frac{1}{2} \log \frac{|\Sigma_{(a+b)}|}{|\Sigma_{(a-b)}|}$$ This formula essentially tells the network: "Make your predictions agree with each other (minimize the variance of the difference), but make sure those predictions actually vary across different images (maximize the variance of the sum) so you aren't just outputting a constant zero."
  4. Competitive IMAX: To handle multiple types of objects, they introduce "responsibilities." Multiple modules compete for an image; the module whose two halves agree most strongly "takes responsibility" for that image and updates its weights accordingly.

Datasets

The paper uses synthetic 2D datasets to validate the theory:

  • Gaussian Blobs: Simple images where an "object" consists of two ends, each made of two Gaussian blobs of intensity. The training set constrained one end to the left half and the other to the right half of the image.
  • Character Features: A more abstract dataset using features from the letters "y" and "g." Instead of raw pixels, the input consisted of pre-extracted parameters (position, orientation, extent) of specific strokes (e.g., the u-shaped curve of a 'y').

Key Insights and Ideas

  • Agreement as a Proxy for Truth: In an unsupervised setting, you don't have a "ground truth" label. The authors' "aha moment" is realizing that consistency between independent parts is a powerful substitute for a label. If two different parts of an image consistently point to the same 4-dimensional coordinate, that coordinate likely represents a meaningful property of the whole object.
  • Discovering Degrees of Freedom: The network isn't told that the 4 output units represent $(x, y, \theta, s)$. It simply discovers that there are four independent ways the parts relate to each other that stay consistent across the dataset.
  • The Z-Score for Recognition: After training, the network can detect if an image contains a "wrong" object or a deformed version of the known object by checking the Z-score of the difference between the two halves' predictions. If they disagree significantly, the object is rejected.

Key Results

  • High Information Capture: In the blob experiment, the network achieved about 7 bits of mutual information, indicating it successfully captured the complex nonlinear relationship between raw pixels and the object's pose.
  • Successful Clustering: In the "Competitive IMAX" experiment, the system was tasked with separating an unlabelled mixture of "y" and "g" images. After a brief "bootstrapping" phase, one module successfully specialized in one shape, winning on 498 cases of its preferred shape and 0 of the other.
  • Linear Efficiency: When the input was already parameterized (the second experiment), the mapping to the object's parameters became linear, allowing the network to converge in just 12 sweeps through the data.

Limitations and Future Work

  • Bootstrapping: The competitive model required a "bootstrap" phase (initially showing modules mostly one type of object) to start specializing. The authors suggest that temporal coherence (assuming images in a sequence are likely the same object) could eliminate this need.
  • Simple Geometry: The experiments were limited to 2D rigid objects. The authors note they hadn't yet tested objects with internal degrees of freedom (like scissors) or 3D projections.
  • Hierarchy: While the model works for two parts, real-world objects have many parts. The authors propose building a hierarchy of modules to recognize increasingly complex objects, though this was not implemented in the paper.

Summary updated

Abstract

Using an unsupervised learning procedure, a network is trained on an en(cid:173) semble of images of the same two-dimensional object at different positions, orientations and sizes. Each half of the network "sees" one fragment of the object, and tries to produce as output a set of 4 parameters that have high mutual information with the 4 parameters output by the other half of the network. Given the ensemble of training patterns, the 4 parameters on which the two halves of the network can agree are the position, orientation, and size of the whole object, or some recoding of them. After training, the network can reject instances of other shapes by using the fact that the predictions made by its two halves disagree. If two competing networks are trained on an unlabelled mixture of images of two objects, they cluster the training cases on the basis of the objects' shapes, independently of the position, orientation, and size.
BibTeX
@inproceedings{zemel1990discovering,
  author = {Richard S. Zemel and Geoffrey E. Hinton},
  title = {{Discovering Viewpoint-Invariant Relationships That Characterize Objects}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
  url = {https://openreview.net/forum?id=HJZwVvZObH},
}
Figure