Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible

Yoshua Bengio, Benjamin Scellier, Olexa Bilaniuk, João Sacramento, Walter SennarXiv 2016 · arXiv

Introduction

Recurrent generative models are powerful tools for understanding sensory data, yet they are notoriously slow. Unlike simple feedforward networks, these models—including Boltzmann machines and Hopfield networks—require multiple iterations to reconcile bottom-up sensory signals with top-down internal priors. This iterative process, known as the "Relaxation Problem," is computationally expensive and biologically problematic; animals must react to environmental stimuli almost instantaneously. This paper explores how neural layers trained as "good auto-encoders" can use a single feedforward pass as a near-perfect initialization, making inference both fast and biologically plausible.

Theoretical Framework: Mutual Prediction

The authors propose a framework where neurons utilize multiple dendritic branches to integrate information. Specifically, they distinguish between:

  • Bottom-up branches: Processing sensory input from lower layers.
  • Top-down branches: Processing contextual priors from higher layers.

The core of the proposed method is the "Good Mutual Prediction" condition. Mathematically, if each dendritic branch is trained to predict the overall somatic activity of the neuron, the network reaches a state where the energy gradient is zero. By training consecutive layers to function as auto-encoders—where feedforward weights act as an "encoder" and feedback weights as a "decoder"—the network achieves harmony between signals. If the decoder perfectly reconstructs the input, the network effectively reaches its fixed point after a single feedforward pass.

Experimental Setup

The theory was evaluated using the MNIST dataset of handwritten digits. The researchers implemented a network with 784 input units and three hidden layers (varying between 500 and 1000 units). The primary metric was the L2 norm of the step size during the relaxation process, representing how much the internal state changed at each iteration.

Key Results

The simulation results demonstrated a significant performance gap between standard initialization and the proposed auto-encoder approach.

Comparison of relaxation step size between random and auto-encoder initializations on linear (left) and log (right) scales.

As shown in the figure above, the "Auto-Encoder" weights led to instant convergence. The update step magnitude dropped to near zero almost immediately, meaning inference was essentially complete after the first step. In contrast, networks with random weights required many more iterations to reach equilibrium. Furthermore, the log-scale plot reveals that auto-encoder weights not only start closer to the solution but also approach the final fixed point at a significantly faster exponential rate.

Biological Plausibility

This model aligns closely with the known anatomy of pyramidal cells in the cerebral cortex. These cells possess distinct basal dendrites for bottom-up input and apical dendrites for top-down input. The paper suggests that local synaptic plasticity mechanisms, such as Spike-Timing-Dependent Plasticity (STDP), naturally train these branches to predict somatic activity. This provides a theoretical bridge between the "Fast Brain" (instinctive feedforward reactions) and the "Deep Brain" (deliberative recurrent reasoning).

Limitations and Future Directions

While promising, several challenges remain:

  • Negative Phase Inference: The current work focuses on the "positive phase" (where targets are known). Achieving similar speed during "free-running" or dreaming states is a separate challenge.
  • Temporal Dynamics: The model currently handles static images like MNIST; its performance in dynamic, changing environments is yet to be determined.
  • Learning Coordination: While the model explains how a trained network functions quickly, a biologically plausible theory for how all layers learn simultaneously without global supervision remains an open question.

Summary updated

Abstract

We consider deep multi-layered generative models such as Boltzmann machines or Hopfield nets in which computation (which implements inference) is both recurrent and stochastic, but where the recurrence is not to model sequential structure, only to perform computation. We find conditions under which a simple feedforward computation is a very good initialization for inference, after the input units are clamped to observed values. It means that after the feedforward initialization, the recurrent network is very close to a fixed point of the network dynamics, where the energy gradient is 0. The main condition is that consecutive layers form a good auto-encoder, or more generally that different groups of inputs into the unit (in particular, bottom-up inputs on one hand, top-down inputs on the other hand) are consistent with each other, producing the same contribution into the total weighted sum of inputs. In biological terms, this would correspond to having each dendritic branch correctly predicting the aggregate input from all the dendritic branches, i.e., the soma potential. This is consistent with the prediction that the synaptic weights into dendritic branches such as those of the apical and basal dendrites of pyramidal cells are trained to minimize the prediction error made by the dendritic branch when the target is the somatic activity. Whereas previous work has shown how to achieve fast negative phase inference (when the model is unclamped) in a predictive recurrent model, this contribution helps to achieve fast positive phase inference (when the target output is clamped) in such recurrent neural models.
BibTeX
@misc{bengio2016feedforward,
  author = {Yoshua Bengio and Benjamin Scellier and Olexa Bilaniuk and João Sacramento and Walter Senn},
  title = {{Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible}},
  year = {2016},
  eprint = {1606.01651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.01651},
}
Figure