Introduction
Back-propagation (BP) is the cornerstone of modern artificial intelligence, yet it remains fundamentally at odds with biological neural systems. Neuroscientists have long identified several "biological sins" in BP: the Weight Transport Problem, where neurons must know the exact transpose of outgoing weights; Global Feedback Pathways, which lead to vanishing gradients in deep networks; and Computational Asymmetry, where forward and backward passes utilize different mathematical operations.
This paper introduces two biologically motivated learning algorithms—LRA-E and DTP-$\sigma$—that address these issues by replacing global error signals with local "target representations." These algorithms allow each layer to update itself based on local discrepancies, achieving stability and generalization that often surpasses back-propagation in deep, nonlinear architectures.
Proposed Methods
The researchers developed two primary algorithms within the "Discrepancy Reduction" framework:
- LRA-E (error-driven Local Representation Alignment): Drawing inspiration from predictive coding, LRA-E utilizes a set of learned "error weights" to project signals from higher layers back to lower ones. Instead of a global derivative, it calculates a local target representation—a version of the layer's activity that would have minimized the observed error. Weights are then updated to align current activity with this target. Notably, LRA-E can operate without the derivatives of activation functions, further increasing its biological plausibility.
- DTP-$\sigma$ (Adaptive Noise Difference Target Propagation): Standard Difference Target Propagation often fails in deep networks due to representation collapse. DTP-$\sigma$ introduces an adaptive noise scheme where the standard deviation of injected noise is dynamically adjusted based on local loss. This coordinates learning across layers and prevents the instabilities common in deep target-propagation models.
Key Insights and Gradient Alignment
A central finding of the study is that learning is highly robust even without perfectly symmetric weights. Unlike BP, which requires exact weight transposes, LRA-E and DTP-$\sigma$ demonstrate that approximate alignment is sufficient for effective optimization. As shown in Figure 1, the gradient angle between these local updates and the true BP gradient stabilizes throughout training, confirming that local rules can effectively approximate the global optimization path.
Experimental Results
The algorithms were evaluated on MNIST and Fashion MNIST using Multi-Layer Perceptrons (MLPs) with 3, 5, and 8 hidden layers.
Stability in Deep Networks
As networks grew deeper, the advantages of local representation alignment became clear. In 8-layer MNIST networks, many popular BP alternatives like Equilibrium Propagation failed to converge. In contrast, LRA-E remained stable, maintaining a strong 2.55% test error. On the more challenging Fashion MNIST dataset, LRA-E consistently outperformed BP across all depths, achieving a 12.31% error rate on a 5-layer network compared to BP’s 12.98%.
Qualitative Representation Analysis
Qualitative analysis using t-SNE visualizations revealed that LRA-E produces more distinct and widely separated clusters of representations in hidden layers compared to other feedback alignment methods. This suggests that the local target-propagation mechanism encourages the network to learn more robust and discriminative features.
Conclusion and Future Work
LRA-E and DTP-$\sigma$ demonstrate that biologically motivated local learning rules are not only plausible but computationally competitive. By breaking global optimization into local alignment problems, these methods avoid the vanishing gradient issues inherent in BP while offering superior generalization in deep, nonlinear networks. Future research will focus on scaling these local target-propagation mechanisms to more complex architectures, such as Convolutional Neural Networks (CNNs) and Residual Networks (ResNets).
Summary updated