Learning Stochastic Feedforward Neural Networks

Yichuan Tang, Ruslan SalakhutdinovNeurIPS 2013 · OpenReview

Standard deep learning models often struggle with a fundamental limitation: the "one-to-many" problem. This paper, published by Yichuan Tang and Ruslan Salakhutdinov from the University of Toronto, introduces a hybrid neural architecture designed to overcome this hurdle. By combining deterministic and stochastic hidden units, their proposed Stochastic Feedforward Neural Networks (SFNNs) can efficiently model complex, multimodal distributions in real-valued data.

The "One-to-Many" Problem

Standard Multilayer Perceptrons (MLPs) act as deterministic function approximators. When used for regression, they typically assume the output follows a unimodal distribution (such as a single Gaussian). In simpler terms, for a given input $X$, the model predicts one "average" output $Y$.

However, many real-world tasks are "structured" or "one-to-many." For example, if a model is shown the top half of a face, there isn't just one "correct" mouth to complete it—there are many possible valid expressions (happy, sad, angry). A standard MLP would average these possibilities into a blurry, nonsensical mess. While models like Sigmoid Belief Nets (SBNs) can handle multimodality, they are notoriously difficult to train on real-valued data and rely on slow sampling methods like Gibbs sampling.

Methods: The Stochastic Feedforward Neural Network

To address these limitations, the authors propose the Stochastic Feedforward Neural Network (SFNN). The architecture is a "directed" model (where information flows one way) but features layers containing two distinct types of units:

  1. Deterministic Units: Standard sigmoid units that provide smooth functional mappings.
  2. Stochastic Units: Binary latent variables that act as "switches," allowing the network to choose between different modes of the output distribution.
SFNN Architecture and Multimodal Mapping

The Training Innovation

The core challenge with stochastic networks is that you cannot simply backpropagate through a random "flip" of a stochastic unit. To solve this, the authors utilize a Generalized EM (Expectation-Maximization) algorithm:

  • E-Step (Inference): They use Importance Sampling to approximate the posterior distribution of the hidden states. They treat the "conditional prior" (the forward pass from the input) as the proposal distribution.
  • M-Step (Learning): They use standard backpropagation to update the weights based on the sampled states.

Crucially, the authors highlight a "cooperative" effect during training: as the model learns, the forward pass (the prior) gets better at picking hidden states that are likely to produce the observed data. This makes the importance sampling increasingly efficient over time.

Datasets and Key Results

The model was evaluated on four distinct types of data, ranging from synthetic benchmarks to complex image datasets.

Synthetic 1D Datasets

The authors tested the model on three toy problems designed to evaluate a model's ability to learn one-to-many mappings, tight modes, and widely separated modes. On these synthetic datasets, SFNNs consistently achieved higher log-probabilities than Mixture Density Networks (MDNs) and Conditional Restricted Boltzmann Machines (C-GRBMs).

SFNN modeling synthetic multimodal distributions

Real-World Image Datasets

  • Toronto Face Database (TFD): Using 4,000 images of faces with different expressions, the task was to model the distribution of expressions given a "mean face." The SFNN achieved a test log-probability of 1534 Nats, significantly outperforming MDNs (1321 Nats) and C-GRBMs (1146 Nats). Furthermore, despite the sampling requirement, SFNNs trained on faces in about 2 minutes, whereas C-GRBMs took over 2.5 hours to reach lower performance.
  • Amsterdam Library of Objects (ALOI): Tested on 1,000 everyday objects under various lighting and rotations, the model successfully generated colorful, realistic textures.
  • Weizmann Horse Database: The model was used to generate realistic horse appearances conditioned purely on a binary shape mask.

Key Insights

  • Distributed Multimodality: In a standard Mixture Density Network (MDN), you must pre-specify $K$ components. In an SFNN, $N$ stochastic binary units can represent up to $2^N$ different modes. This "distributed" representation is vastly more parameter-efficient.
  • The Hybrid Advantage: The paper demonstrates that purely stochastic models struggle with continuous data. Adding deterministic units is "a big win," allowing the model to learn smooth mappings and small residual variances that stochastic units alone cannot capture.
  • Robustness to Noise: Because the model learns a rich latent representation, it excels at denoising. By inferring the "expected" hidden state from a corrupted image, the model can reconstruct missing pixels more accurately than simple averaging. Using these latent features also improved the accuracy of facial expression classifiers, especially under 50% occlusion or heavy noise.

Limitations and Future Work

While highly effective, the SFNN approach does have limitations. The sampling overhead means the model still requires drawing $M$ samples (the authors used 20-30) for every weight update, making it more computationally expensive than a standard MLP. Additionally, at test time, calculating the exact probability $p(y|x)$ remains intractable and requires Monte Carlo approximation.

As a future direction, the authors suggested learning a separate "recognition network" to perform inference in a single forward pass. This insight closely foreshadowed the development of Variational Autoencoders (VAEs), which would emerge shortly after this work and revolutionize the field of generative modeling.

Summary updated

Abstract

Multilayer perceptrons (MLPs) or neural networks are popular models used for nonlinear regression and classification tasks. As regressors, MLPs model the conditional distribution of the predictor variables Y given the input variables X. However, this predictive distribution is assumed to be unimodal (e.g. Gaussian). For tasks such as structured prediction problems, the conditional distribution should be multimodal, forming one-to-many mappings. By using stochastic hidden variables rather than deterministic ones, Sigmoid Belief Nets (SBNs) can induce a rich multimodal distribution in the output space. However, previously proposed learning algorithms for SBNs are very slow and do not work well for real-valued data. In this paper, we propose a stochastic feedforward network with hidden layers having \emph{both deterministic and stochastic} variables. A new Generalized EM training procedure using importance sampling allows us to efficiently learn complicated conditional distributions. We demonstrate the superiority of our model to conditional Restricted Boltzmann Machines and Mixture Density Networks on synthetic datasets and on modeling facial expressions. Moreover, we show that latent features of our model improves classification and provide additional qualitative results on color images.
BibTeX
@inproceedings{tang2013learning,
  author = {Yichuan Tang and Ruslan Salakhutdinov},
  title = {{Learning Stochastic Feedforward Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
  url = {https://openreview.net/forum?id=By4F-t-_-r},
}
Figure