TL;DR
Feedback Alignment (FA) offers a biologically plausible alternative to backpropagation by using fixed random weights for the backward pass, but it notoriously fails to scale to deep networks. This paper discovers that deep FA suffers from "rank collapse," where gradient updates become trapped in a highly restricted, low-dimensional subspace. By introducing geometric interventions like the Muon optimizer or Batch Normalization, the authors successfully counteract this collapse, rescuing FA performance in deep convolutional networks.
What Problem Does This Paper Address?
In modern deep learning, backpropagation (BP) is the undisputed king. However, neuroscientists have long pointed out that BP is biologically implausible due to the "weight transport problem": it requires backward error-propagating synapses to share the exact same weights as forward synapses.
Feedback Alignment (FA) bypassed this issue by showing that networks can learn even if the backward pass uses fixed, random matrices. In shallow networks, the forward weights naturally "align" themselves to these random backward projections. Unfortunately, FA completely falls apart in deeper architectures like convolutional neural networks (CNNs). This paper tackles a fundamental question: What exactly stops weight alignment from happening in deep networks?
Methods
The authors first diagnose the geometry of FA gradients by measuring their effective rank (a continuous approximation of dimensionality). They observe a clear pathology: as deep FA networks attempt to train, their gradient trajectories suffer from severe rank collapse.
To combat this low-dimensional bottleneck, they evaluate two distinct mechanisms designed to inject dimensionality back into the training process:
- The Muon Optimizer: Unlike standard SGD, Muon applies an orthogonalization step (via Singular Value Decomposition) to the momentum matrix, effectively flattening its singular value spectrum (setting all non-zero singular values to 1). This forces the optimizer to explore lesser-used parameter directions.
- Batch Normalization (BN): Applied to hidden layer activations, BN has been shown to promote activation orthogonality, preventing the internal representations from collapsing into low-dimensional subspaces.
Additionally, they experiment with adding a local decorrelation loss to minimize channel covariance and use the Freon optimizer to smoothly interpolate between standard SGD and Muon geometry.
Datasets
The approaches were evaluated on four standard computer vision benchmarks of increasing difficulty:
- CIFAR-10 (used for initial diagnostic analysis and depth scaling tests)
- STL-10
- CIFAR-100
- Tiny ImageNet (a 200-class subset of ImageNet downsampled to 64x64)
Key Insights and Ideas
- Rank Collapse is the Core Obstacle: In deep networks trained with standard optimizers, FA gradients quickly become highly correlated across successive steps. The network gets stuck updating the same few directions, preventing deeper layers from ever achieving alignment.
- Pure Noise is Not the Answer: One might think adding random noise would increase dimensionality and fix the issue. However, the authors show that adding isotropic Gaussian noise actually degrades performance (see Figure 7 in the paper). The dimensionality expansion must be structurally aligned with the true learning signal.
- Complementary Geometric Interventions: Muon alters the geometry of the parameter updates, while Batch Normalization alters the geometry of the activation space. Because they attack rank collapse from two different angles, combining them yields a powerful synergistic effect.
Key Results
The geometric interventions successfully prevented the typical performance degradation seen in FA as networks get deeper.
When scaling up to deeper models like ResNet-18 and AlexNet, the results were stark (Table 1):
- ResNet-18 on CIFAR-100: Baseline FA completely fails, achieving a dismal 1.4% accuracy. Introducing both Muon and Batch Normalization rescues the network, boosting accuracy to 46.1%.
- ResNet-18 on STL-10: Baseline FA achieves 14.5%, which skyrockets to 64.2% when Muon and BN are combined.
- AlexNet on Tiny ImageNet: Baseline FA yields 6.4%, which increases to 32.1% with the combined approach.
Limitations and Future Work
While this work provides a major stepping stone for alternative credit-assignment strategies, a few key limitations remain:
- The Backprop Gap: Even with Muon and BN, a significant performance gap still exists between FA and standard backpropagation on difficult tasks (e.g., 46.1% vs 75.2% for ResNet-18 on CIFAR-100).
- Biological Plausibility of the Fixes: Neither Muon (which requires global SVD operations) nor standard Batch Normalization (which relies on global batch statistics) are biologically plausible.
The authors suggest that future work should focus on finding local, biologically realistic approximations to these mechanisms—such as Oja's rule for local principal component extraction or divisive normalization in neural circuits—to achieve high-dimensional update dynamics in a truly brain-like fashion.
Summary updated