Abstract
This paper introduces random Contrastive Hebbian Learning (rCHL), a biologically plausible alternative to backpropagation that eliminates the need for symmetric synaptic weights. By replacing the mathematically convenient but biologically unrealistic transpose weight matrix with fixed random feedback weights, the authors demonstrate that neural networks can still learn complex tasks like MNIST classification and autoencoding with performance comparable to standard gradient-based methods.
Introduction
The dominant algorithm for training deep neural networks is gradient backpropagation (BP). However, BP faces the "weight transport problem": it requires the feedback weights used during the backward pass to be the exact transpose of the forward weights. In the brain, synapses are generally unidirectional, and there is no known mechanism for a "master controller" to communicate the exact strength of one synapse to another distant one to maintain symmetry.
Additionally, BP often lacks temporal dynamics and requires the precise calculation of derivatives for every layer. While Contrastive Hebbian Learning (CHL)—which learns via the difference between a "free" phase and a "clamped" phase—is more biologically grounded, it traditionally still relies on symmetric weights. This paper bridges this gap by proving that CHL can function effectively even when the feedback path is entirely random and fixed.
Methods
The authors propose rCHL, which modifies the standard CHL framework by replacing the transpose matrix ($W^T$) with a fixed random matrix ($G$) to propagate signals backward during the "clamped" phase.
In this framework, the network is treated as a dynamical system described by non-linear differential equations that settles into equilibrium in two distinct phases:
- Clamped Phase: The output layer is fixed to the target label.
- Free Phase: The output layer settles based only on the input.
Weights are updated using a Hebbian rule based on the difference between the correlations of pre- and post-synaptic activity in these two phases: $\Delta W_k \propto (\hat{x}_k \otimes \hat{x}_{k-1} - \check{x}_k \otimes \check{x}_{k-1})$, where $\hat{x}$ represents clamped activity and $\check{x}$ represents free activity.
Results and Performance
The algorithm was evaluated on several benchmarks, including XOR, MNIST, and a synthetic "Bars and Stripes" dataset. On MNIST, rCHL achieved 97.5% accuracy, nearly identical to standard CHL (97.7%) and superior to the authors' BP baseline (96.2%).
As shown in the synthetic Bars and Stripes task, rCHL convergence (both in terms of MSE and Accuracy) closely tracks that of symmetric CHL. The forward weights eventually "align" themselves to work with the random feedback signal. On the more challenging eMNIST (letters) task, rCHL reached 84.6% accuracy, demonstrating robust learning without weight symmetry, though slightly lower than the symmetric version (89.9%).
Key Insights and Discussion
- Distribution Matters: Pseudospectra analysis revealed that random matrices drawn from a Normal distribution lead to faster convergence than those from a Uniform distribution by shifting the pseudospectrum toward the right-half complex plane.
- Feedback Gain ($\gamma$): There is a critical trade-off with the feedback gain parameter. As the number of layers increases, $\gamma$ must be increased to ensure the teaching signal reaches the earliest layers.
- Biological Mapping: rCHL can be viewed as a model for top-down signaling in the cortex, where higher-level cognitive areas provide a "clamping" signal to primary sensory areas.
Limitations and Future Work
While rCHL shows promise, a performance gap remains on high-dimensional datasets like eMNIST. Furthermore, the model still requires two distinct phases (free and clamped), whereas a truly biological model would likely operate in a single, continuous stream of time. Future research aims to implement rCHL in spiking neural networks and asynchronous update schemes to increase compatibility with neuromorphic hardware.
Summary updated