Introduction
In the early 1990s, unsupervised learning algorithms excelled at extracting low-level features, such as edges, from images. However, they consistently struggled to organize these rudimentary features into high-level, position-invariant concepts—like recognizing a "triangle" regardless of where it appears in the visual field.
A popular strategy to bridge this gap was to maximize "coherence." This principle assumes that if you observe two nearby patches of an image, or two successive frames of a video, they should convey the same underlying information. While promising, early methods like Imax were often too rigid. They attempted to force a perfect, one-to-one agreement between inputs. While strict agreement works for final, high-level classifications, it fails for intermediate features. For instance, an edge at one location in a video frame does not perfectly predict an identical edge at the exact same location in the next frame of a moving object; rather, it predicts a range of possible nearby edges.
JPMAX addresses this limitation by treating object recognition as a model-fitting problem, allowing for a more flexible, "probabilistic" agreement over time. By forcing a neural network's output to match a "prior" joint probability distribution, the system learns to group different views of the same object together, achieving position-invariant recognition without relying on any labeled data.
The JPMAX Architecture and Approach
The core innovation of JPMAX is the shift from strict, one-to-one feature agreement to probabilistic model-fitting.
The architecture relies on a network divided into two groups of units. These groups receive inputs that are related but physically or temporally separated—for example, the same area of a retina at two different time steps, $t$ and $t+1$. Each unit in these groups employs a "softmax" activation function, meaning its output represents the probability that the given input belongs to a specific learned category.
Relaxing Constraints with Block-Diagonal Priors
Instead of forcing the two groups of units to always output the exact same category (a strict diagonal relationship), JPMAX defines a prior joint distribution ($p_{ij}$). This matrix describes the desired relationship between the units across time.
The "aha moment" of the paper is the realization that intermediate layers shouldn't be forced into perfect agreement. By utilizing a "block-diagonal" prior, the network is permitted to have multiple units representing different "views" or "versions" of the same object class. For example, a block-diagonal prior dictates: "If Group A activates a unit in Block 1, Group B should also activate a unit in Block 1, but it doesn't matter which specific unit within that block."
During training, the network calculates the actual joint distribution ($q_{ij}$) of its outputs over a training set. It then uses a learning rule—based on Kullback-Leibler divergence or a multinomial approximation—to adjust its weights so that the actual distribution $q$ closely matches the target prior $p$.
Experimental Setup
To test the system's ability to learn translation invariance, the authors utilized a synthetic dataset:
- Geometric Objects: 1,500 training patterns consisting of simple shapes (circles, squares, and triangles).
- The Grid: Objects were rendered on a 12x12 pixel grid.
- Motion Simulation: To simulate a moving observer, the same object appears for 10 successive frames, but its center "jitters" randomly within a 6x6 central patch.
- Overlaps: To introduce complexity, some sequences featured ambiguous "overlapping" frames where a new object appeared before the old one fully disappeared.
Key Results and Insights
By using temporal coherence as a teacher, the network successfully learned what a shape was simply because the pixels belonging to that shape tended to move together over time. It required no explicit labels, only the observation of motion.
- Feature Discovery: The network demonstrated layer-by-layer learning capabilities. In the first layer, units successfully specialized into distinct feature detectors. One block of units learned to detect circles in various positions, another specialized in horizontal and vertical lines (useful for squares), and a third focused on blurred triangular shapes.
- Position Invariance: After freezing the first layer and training a second layer on top of it, the network achieved accurate, position-invariant classification. Even as the objects jittered around the 12x12 grid, the final output units reliably and consistently identified "Circle," "Square," and "Triangle."
- Convergence: The learning process proved to be stable, typically converging within 1,000 to 3,000 iterations of steepest descent, overcoming the local optima problems that frequently plagued earlier back-propagation methods.
Limitations and Future Work
While successful on simple datasets, JPMAX has notable limitations. The network struggled with "ambiguous" patterns; when multiple objects overlapped, it typically only detected one of the objects or occasionally misclassified them entirely.
Furthermore, while the authors acknowledge that using more complex priors for hidden layers could theoretically improve results, doing so would significantly increase the mathematical complexity of the learning rule. Finally, scaling this approach to real-world images remains a challenge. The authors suggest that for complex, real-world objects sharing low-level features (like corners), a deeper architecture with at least three layers of weights would be necessary to progress from simple parts, to viewpoint-specific features, and finally to true viewpoint invariance.
Summary updated