A theory of local learning, the learning channel, and the optimality of backpropagation

Pierre Baldi, Peter J. Sadowski

Introduction

For decades, neural network research has oscillated between the biological plausibility of Hebbian learning and the engineering success of backpropagation (BP). While Hebbian learning—often summarized as "neurons that fire together, wire together"—is biologically inspired, it has historically lacked a precise mathematical definition, making it difficult to analyze its limitations or compare it to BP. This paper formalizes the concept of local learning rules and introduces the "learning channel" framework to demonstrate that backpropagation is not merely a popular algorithm, but a mathematically optimal one for information transfer.

Formalizing Local Learning

The authors categorize learning rules based on the principle of locality. A rule is defined as local if the change in a synaptic weight ($\Delta w_{ij}$) depends only on variables available at that specific synapse, such as pre-synaptic activity, post-synaptic activity, and the weight itself.

Table of common learning rules and their mathematical expressions.

By classifying these rules through polynomial stratification (e.g., quadratic vs. cubic), the authors analyze their stability and convergence. A key insight is that "local" and "unsupervised" are distinct concepts; a supervised rule like the Delta rule can be local if the target is provided directly to the unit. The fundamental challenge in deep learning is not locality itself, but the communication of target information to deep layers.

The Learning Channel and Optimality

The paper conceptualizes the flow of error information from the output back to deep weights as a learning channel. This communication protocol is analyzed using information-theoretic tools to define the "Capacity" of a learning algorithm—the number of bits of gradient information provided per weight relative to the computational cost.

Commutative diagram illustrating the relationship between representations and transformations in the learning process.

Backpropagation is shown to be a highly efficient communication protocol. It transmits the maximum possible information about the gradient ($D$ bits) at the minimum possible computational cost (roughly one forward-pass equivalent). The authors prove that BP achieves the theoretical maximum for both the rate of information transfer and the expected improvement per step, making it mathematically superior to alternatives like weight or activity perturbation.

Algorithm Info Rate ($\mathcal{R}$) Improvement ($\mathcal{O}$) Efficiency
Backpropagation (BP) $D$ 1 Optimal
Weight Perturbation $1/W$ $C/\sqrt{W}$ Extremely Low
Activity Perturbation $D/N$ 1 Low

Results and Empirical Validation

The theoretical framework was validated using several benchmarks:

  • MNIST: Used to simulate various learning rules and test the "Deep Targets" algorithm—a novel approach for training non-differentiable networks by sampling potential hidden layer targets.
  • Boolean Functions: Analysis of all possible 2- and 3-variable Boolean functions demonstrated the fundamental limitations of stacking purely local, unsupervised rules.
  • Non-differentiable Networks: The Deep Targets algorithm successfully trained threshold networks on MNIST-style clusters, achieving low error where standard BP fails due to zero gradients.

Conclusion

The paper provides a rigorous "upper bound" for learning efficiency. It suggests that any physical system, biological or artificial, that aims to learn complex tasks efficiently must implement a mechanism mathematically equivalent to backpropagation. While biological systems may use asymmetric feedback paths, the underlying information-theoretic optimality of the learning channel remains a requirement for high-performance learning.

Summary updated

Abstract

In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. We estimate the learning channel capacity associated with several algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost, even in recurrent networks. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.
BibTeX
@article{baldi2016theory,
  author = {Pierre Baldi and Peter J. Sadowski},
  title = {{A theory of local learning, the learning channel, and the optimality of backpropagation}},
  journal = {Neural Networks},
  year = {2016},
  doi = {10.1016/j.neunet.2016.07.006},
}
Figure