Learning Generative Models with the Up-Propagation Algorithm

Jong-Hoon Oh, H. Sebastian SeungNeurIPS 1997

TL;DR

This paper introduces "up-propagation," a novel algorithm for learning generative models in neural networks. Unlike traditional backpropagation which focuses on recognition, up-propagation learns to generate data by inverting a generative model, effectively performing a "generate-test-revise" cycle. This approach shows promise in learning complex, nonlinear pattern manifolds, outperforming Principal Component Analysis (PCA) in reconstruction tasks for handwritten digits.

What Problem Does This Paper Address?

The paper tackles the challenge of learning generative models, which aim to understand and reproduce the underlying data distribution. While backpropagation is a powerful tool for training recognition models (learning to classify or analyze input), it doesn't directly address how to build models that can create new data instances. The authors draw inspiration from Helmholtz's doctrine of "unconscious inference," proposing that perception involves generating hypotheses (top-down) and then refining them based on sensory data (bottom-up). They aim to implement this idea in neural networks, creating an algorithm that can both generate patterns and learn the generative process from examples.

Methods

The core innovation is the up-propagation algorithm. It works by:

  • Generative Model: A layered neural network where top-down connections generate patterns from hidden variables.
  • Inversion (Recognition/Analysis): Given sensory input, the algorithm iteratively refines a hypothesis about the hidden variables. This is achieved through a negative feedback loop:
  • A top-down pass generates a pattern.
  • A bottom-up pass calculates an error signal based on the difference between the generated pattern and the input.
  • This error signal is propagated upwards to adjust the hidden variables, leading to a revised hypothesis. This process is framed as gradient descent on the reconstruction error.
  • Learning: The same error signal used for inversion is also used to update the synaptic weights of the generative model. This allows the network to learn to generate patterns that better match the training data.

Up-propagation is presented as a generalization of Principal Component Analysis (PCA) to nonlinear, multilayer generative models.

Datasets

  • USPS database: This dataset of handwritten digits was used for training and evaluating the up-propagation algorithm. The authors note that they trained separate models for each digit class.

Key Insights and Ideas

  • Generative Models as Hypothesis Generators: The paper frames generative models as systems that propose hypotheses about data, which are then tested and refined against actual sensory input.
  • Unified Inversion and Learning: The elegant insight is using the same error signal for both inferring hidden variables (recognition/analysis) and learning the model's parameters.
  • Nonlinear Pattern Manifolds: Up-propagation can learn complex, nonlinear relationships in data, creating a "pattern manifold" that is more flexible than the linear approximations offered by PCA.
  • Iterative Inversion is Fast and Effective: Despite potential concerns about local minima, the iterative inversion process proved surprisingly fast and yielded good quality reconstructions.

Key Results

  • Superior Reconstruction: Up-propagation networks achieved significantly lower reconstruction errors compared to PCA for handwritten digits. For example, a 256-12 up-prop network performed as well as PCA with 36 principal components.
  • Learning Nonlinear Models: The algorithm successfully learned nonlinear generative models that captured the variations within digit classes better than linear PCA models.
  • One-Step vs. Two-Step Generation: The paper explored both one-step (L=1) and two-step (L=2) generative models. While two-step models offered richer representation, they also required longer training times.

Limitations and Future Work

  • Recognition Performance: While the generative models performed well, using them directly for classification (by inverting all models and picking the best fit) was not encouraging. The nonlinearity improved generation for all classes, not just the correct one, hindering clear classification.
  • Single Model for All Digits: The authors suggest training a single generative model for all digits, with some hidden variables representing the digit class, as a future direction for improved recognition.
  • Deterministic Model: The core generative model is deterministic, which limits its ability to model inherent pattern variability. The authors suggest that adding probabilistic elements could address this.
  • Permutation Symmetry: For deeper networks (e.g., 256-25-9), learning could be slow due to permutation symmetry, suggesting potential architectural or optimization challenges.
  • Indirect Recognition: Iterative inversion for recognition is considered a rather indirect approach, hinting at a need for more direct recognition methods.

Summary updated

BibTeX
@inproceedings{oh1997learning,
  author = {Jong-Hoon Oh and H. Sebastian Seung},
  title = {{Learning Generative Models with the Up-Propagation Algorithm}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}

Loading related content...

Figure