Learning with augmented target information: An alternative theory of Feedback Alignment

Huzi Cheng, Joshua W. BrownarXiv 2023 · DOI · arXiv

Abstract

Feedback Alignment (FA) has long been considered a biologically plausible alternative to Backpropagation (BP), typically interpreted as an approximation of the gradient. This paper challenges that view, proposing that FA works by acting as a communication channel that embeds target information directly into hidden layers. By reframing the neural network as an encoder-decoder system, the authors show that FA variants can successfully train networks by making hidden representations more "predictable" for the rest of the architecture. This theory explains FA's success in scenarios where gradient-based theories fail, such as in deep non-linear networks or when output weights are frozen.

Introduction: The Weight Symmetry Problem

The standard method for training deep neural networks, Backpropagation (BP), relies on the "Gold Standard" of gradient descent. However, BP faces a significant hurdle in biological plausibility known as the Weight Symmetry Problem. To compute weight updates for a specific neuron, the brain would need to know the exact weights of all downstream connections. This requirement for "non-local" information sharing is highly unlikely in biological neural circuits.

Feedback Alignment (FA) emerged as a solution by replacing downstream weights with a fixed random matrix. While FA is effective, the prevailing explanation—that random weights eventually "align" with true gradients—struggles to account for its performance in complex, non-linear architectures. This research seeks a more robust, architecture-agnostic explanation rooted in information theory.

A New Theoretical Framework: The Encoder-Decoder View

The authors move away from calculus-based gradient approximation, instead framing the neural network as two distinct components: an Encoder (mapping input $x$ to hidden layer $h$) and a Decoder (mapping hidden layer $h$ to output $y$). In this view, the feedback path is not a derivative calculator but a communication channel that augments the hidden state with information about the target $Y$.

Conceptual diagram of the proposed encoder-decoder framework with a feedback circuit for target information propagation.

To test this "Information Augmentation" theory, the researchers introduced several novel FA variants:

  • Noisy Feedback Alignment (NFA): Injects Gaussian noise into feedback weights during every update, simulating a noisy channel.
  • Network Feedback (NF): Replaces the feedback matrix with a fixed, non-linear neural network.
  • Target Feedback (TF): Propagates raw target information ($Y$) directly, bypassing the error signal entirely.

Experimental Setup and Datasets

The theory was evaluated across several benchmarks designed to measure information flow:

  • IBTask (Information Bottleneck Task): A binary classification task on a 2D sphere.
  • MNIST: The standard handwritten digit classification dataset.
  • Regression Variants: Both IBTask and MNIST were adapted into regression problems (IBRegTask and MNISTRegTask) to test the limits of the Target Feedback approach.

Additionally, the authors utilized Scalable Activation Functions by adding a tunable scaling factor ($\gamma$) to neurons. This "$\gamma$ trick" allows hidden states to explore a larger space without sacrificing non-linearity, helping FA overcome "dead zones" where it might otherwise stall compared to BP.

Key Results and Insights

The experiments yielded several striking findings that support the information-theoretic view of FA over simple gradient approximation.

Comparable Performance and Robustness

The proposed variants (NFA, NF, and TF) achieved performance levels comparable to standard FA and BP across both classification and regression tasks. Most notably, Target Feedback (TF) successfully trained networks for regression despite ignoring the current error signal. This suggests that an explicit error signal may not be strictly necessary for learning if sufficient target information is provided to the hidden layers.

Performance comparison of FA and BP variants on classification and regression tasks, showing robustness even when output weights are frozen ($\Delta W_O$ OFF).

Learning with Frozen Output Weights

As shown in Figure 1, FA and its variants can improve the "predictability" of hidden layers even when the output weights are frozen ($\Delta W_O$ OFF). This is a critical piece of evidence: it suggests the hidden layer learns to align its representations with the target independently of the final layer's specific configuration. This behavior is difficult to explain via gradient alignment but fits perfectly within the information augmentation framework.

Representational Drift

The success of NFA provides a potential computational model for "representational drift" observed in neuroscience. In biological brains, internal activity patterns often change over time even when the information they represent remains stable. NFA demonstrates that stable learning can occur even through a constantly changing, noisy feedback channel, provided the channel preserves target information.

Conclusion and Future Directions

This paper provides a compelling alternative to the gradient-approximation theory of Feedback Alignment. By viewing feedback as a means of target information augmentation, we gain a more flexible understanding of how biological and artificial systems might learn.

Future work will explore how target dimensionality affects learning and whether more complex feedback pathways, such as Variational Autoencoders, can further enhance the efficiency of information embedding in deep networks. While calculating Mutual Information in high-dimensional settings remains a challenge, this framework opens new doors for biologically plausible deep learning.

Summary updated

Abstract

While error backpropagation (BP) has dominated the training of nearly all modern neural networks for a long time, it suffers from several biological plausibility issues such as the symmetric weight requirement and synchronous updates. Feedback Alignment (FA) was proposed as an alternative to BP to address those dilemmas and has been demonstrated to be effective on various tasks and network architectures. Despite its simplicity and effectiveness, a satisfying explanation of how FA works across different architectures is still lacking. Here we propose a novel, architecture-agnostic theory of how FA works through the lens of information theory: Instead of approximating gradients calculated by BP with the same parameter, FA learns effective representations by embedding target information into neural networks to be trained. We show this through the analysis of FA dynamics in idealized settings and then via a series of experiments. Based on the implications of this theory, we designed three variants of FA and show their comparable performance on several tasks. These variants also account for some phenomena and theories in neuroscience such as predictive coding and representational drift.
BibTeX
@misc{cheng2023learning,
  author = {Huzi Cheng and Joshua W. Brown},
  title = {{Learning with augmented target information: An alternative theory of Feedback Alignment}},
  year = {2023},
  eprint = {2304.01406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.01406},
}
Figure