Introduction
For decades, neural network research has oscillated between the biological plausibility of Hebbian learning and the engineering success of backpropagation (BP). While Hebbian learning—often summarized as "neurons that fire together, wire together"—is biologically inspired, it has historically lacked a precise mathematical definition, making it difficult to analyze its limitations or compare it to BP. This paper formalizes the concept of local learning rules and introduces the "learning channel" framework to demonstrate that backpropagation is not merely a popular algorithm, but a mathematically optimal one for information transfer.
Formalizing Local Learning
The authors categorize learning rules based on the principle of locality. A rule is defined as local if the change in a synaptic weight ($\Delta w_{ij}$) depends only on variables available at that specific synapse, such as pre-synaptic activity, post-synaptic activity, and the weight itself.
By classifying these rules through polynomial stratification (e.g., quadratic vs. cubic), the authors analyze their stability and convergence. A key insight is that "local" and "unsupervised" are distinct concepts; a supervised rule like the Delta rule can be local if the target is provided directly to the unit. The fundamental challenge in deep learning is not locality itself, but the communication of target information to deep layers.
The Learning Channel and Optimality
The paper conceptualizes the flow of error information from the output back to deep weights as a learning channel. This communication protocol is analyzed using information-theoretic tools to define the "Capacity" of a learning algorithm—the number of bits of gradient information provided per weight relative to the computational cost.
Backpropagation is shown to be a highly efficient communication protocol. It transmits the maximum possible information about the gradient ($D$ bits) at the minimum possible computational cost (roughly one forward-pass equivalent). The authors prove that BP achieves the theoretical maximum for both the rate of information transfer and the expected improvement per step, making it mathematically superior to alternatives like weight or activity perturbation.
| Algorithm | Info Rate ($\mathcal{R}$) | Improvement ($\mathcal{O}$) | Efficiency |
|---|---|---|---|
| Backpropagation (BP) | $D$ | 1 | Optimal |
| Weight Perturbation | $1/W$ | $C/\sqrt{W}$ | Extremely Low |
| Activity Perturbation | $D/N$ | 1 | Low |
Results and Empirical Validation
The theoretical framework was validated using several benchmarks:
- MNIST: Used to simulate various learning rules and test the "Deep Targets" algorithm—a novel approach for training non-differentiable networks by sampling potential hidden layer targets.
- Boolean Functions: Analysis of all possible 2- and 3-variable Boolean functions demonstrated the fundamental limitations of stacking purely local, unsupervised rules.
- Non-differentiable Networks: The Deep Targets algorithm successfully trained threshold networks on MNIST-style clusters, achieving low error where standard BP fails due to zero gradients.
Conclusion
The paper provides a rigorous "upper bound" for learning efficiency. It suggests that any physical system, biological or artificial, that aims to learn complex tasks efficiently must implement a mechanism mathematically equivalent to backpropagation. While biological systems may use asymmetric feedback paths, the underlying information-theoretic optimality of the learning channel remains a requirement for high-performance learning.
Summary updated