The spectacular success of modern deep learning rests almost entirely on Backpropagation (BP). To train a network with billions of parameters, BP computes a global loss at the output layer and systematically funnels error gradients backward through the network using the chain rule.
However, this global mechanism introduces severe architectural bottlenecks. First, it requires weight transport symmetry: the backward pass must use the exact transpose of the forward weight matrices. Second, it causes backward locking: early layers cannot update their weights until the entire forward pass completes and the global error signal travels all the way back.
In contrast, biological brains learn continuously and efficiently without a centralized clock or global feedback loops. A biological synapse only has access to local information—the electrical and chemical activity of its immediate pre- and post-synaptic neurons. This stark discrepancy has inspired a major research direction: replacing global backpropagation with local credit assignment strategies that coordinate global learning through decentralized, local interactions.
Predictive Coding: Error Propagation via Local Relaxation
One of the most prominent frameworks for local learning is Predictive Coding (PC). Instead of treating layers as passive feature extractors that wait for a backward error signal, PC transforms every layer into an active predictor.
In a PC network, each layer attempts to predict the activity of the layer below it. If the prediction is imperfect, a local prediction error is generated. Learning proceeds by adjusting the neural activities and synaptic weights to minimize these local errors simultaneously.
Standard Backprop:
[Input] ---> [Layer 1] ---> [Layer 2] ---> [Output] ---> (Global Loss)
^ | | |
└------------┴--------------┴--------------┴-- [Global Error Pass]
Predictive Coding:
[Input] <---> [Layer 1] <---> [Layer 2] <---> [Output]
^ ^ ^ ^
(Local Error) (Local Error) (Local Error) (Local Error)
Why does this local minimization lead to global optimization? As demonstrated by Song et al., when a predictive coding network is allowed to settle into an equilibrium state, the local prediction errors implicitly carry the exact same gradient information as a standard backward pass. PC can exactly replicate BP on arbitrary architectures without ever explicitly invoking the chain rule.
Crucially, PC is more than just a biological approximation of BP. Innocenti et al. proved that PC naturally functions as an adaptive, second-order trust-region method. By leveraging local curvature information (Fisher information), PC alters the optimization landscape, allowing deep networks to escape saddle points significantly faster than standard gradient descent. To scale this mechanism to modern deep architectures, Innocenti et al. introduced $\mu$PC, a parameterization scheme that successfully scales predictive coding to networks with over 100 layers while enabling zero-shot transfer of optimal learning rates.
Equilibrium Propagation: Energy Minimization as Gradient Descent
Another elegant approach to bypassing the weight transport problem is Equilibrium Propagation (EP). Designed for energy-based recurrent networks, EP eliminates the need for a separate backward circuit by using the network's own physical dynamics to propagate error signals.
EP operates in two distinct phases:
- The Free Phase: The network receives an input, and its neurons interact recurrently until the entire system settles into a state of minimal energy (a steady-state equilibrium).
- The Nudging Phase: The output neurons are slightly "nudged" toward the correct target value. This local perturbation introduces a wave of adjustments that ripples backward through the recurrent connections, altering the equilibrium state of the hidden layers.
The intuition is beautiful: the difference between the nudged equilibrium state and the free equilibrium state provides the exact gradient of the loss function. Ernoult et al. proved that in discrete-time recurrent networks with static inputs, these local EP updates are mathematically identical to the gradients computed by Backpropagation Through Time (BPTT).
Historically, energy-based relaxation was slow because the network had to settle twice for every training step. To resolve this computational bottleneck, Gavves et al. proposed Initialized Equilibrium Propagation, which trains a fast feedforward network to instantly predict the steady-state behavior of the energy system, drastically accelerating training.
Forward-Only and Blockwise Learning
If propagating errors backward—even via local relaxation—is problematic, can we eliminate the backward pass entirely? Forward-only and blockwise learning algorithms answer this by dividing the network into independently optimized segments.
Instead of optimizing an end-to-end global loss, the network is split into discrete blocks, each equipped with its own local objective function. For example, Siddiqui et al. demonstrated that individual blocks of a deep ResNet-50 can be trained completely independently using local self-supervised losses (such as the Barlow Twins objective). This blockwise approach achieves a top-1 accuracy on ImageNet within 1.1% of standard end-to-end backpropagation, proving that deep hierarchies can emerge without global gradient coordination.
Taking this a step further, the Cascaded Forward (CaFo) algorithm developed by Zhao et al. trains independent neural blocks sequentially using local predictors, removing the need for complex negative sampling or backward passes entirely. Similarly, PEPITA, a rule explored by Dellaferrera et al., utilizes a two-pass forward approach: the first pass computes a standard inference, and the second pass modulates the original input with the output error, driving local weight updates without ever transporting weights or freezing forward execution.
Hardware Acceleration and Neuromorphic Efficiency
While local credit assignment offers profound insights into cognitive neuroscience, its most immediate engineering impact is on neuromorphic and analog hardware.
Standard backpropagation is highly inefficient on distributed hardware because every layer must store its activations in memory and wait for the global backward pass to finish. Local learning rules unlock massive spatial and temporal parallelism because layers can update their weights asynchronously.
Backpropagation Parallelism Bottleneck:
Forward: [Layer 1] ---> [Layer 2] ---> [Layer 3] ---> [Loss]
|
Backward: [Layer 1] <--- [Layer 2] <--- [Layer 3] <------┘
(Locked) (Locked) (Active)
Local Learning Parallelism:
[Layer 1] (Computes local update asynchronously)
[Layer 2] (Computes local update asynchronously)
[Layer 3] (Computes local update asynchronously)
This hardware compatibility is especially powerful when paired with Spiking Neural Networks (SNNs), which communicate via discrete, energy-efficient pulses. For instance, Karlsson et al. introduced Difference Predictive Coding (DiffPC), which transmits local prediction errors only when they exceed a specific threshold. By communicating via sparse, ternary spikes rather than continuous floating-point values, DiffPC reduces data movement by over two orders of magnitude while maintaining high accuracy.
Furthermore, deploying these algorithms on real-world analog chips requires resilience to hardware noise and manufacturing imperfections. To address this, Høier et al. developed $DP^\top$, a variant of Dual Propagation that remains numerically stable even under asymmetric nudging and noisy hardware constraints. By grounding global optimization in local, physically realizable dynamics, this research direction paves the way for a new generation of autonomous, ultra-low-power AI hardware that learns continuously from its environment.