Overcoming Rank Collapse in Feedback Alignment

Gauthier Boeshertz, Razvan Pascanu, Claudia ClopatharXiv 2026 · arXiv

TL;DR

Feedback Alignment (FA) offers a biologically plausible alternative to backpropagation by using fixed random weights for the backward pass, but it notoriously fails to scale to deep networks. This paper discovers that deep FA suffers from "rank collapse," where gradient updates become trapped in a highly restricted, low-dimensional subspace. By introducing geometric interventions like the Muon optimizer or Batch Normalization, the authors successfully counteract this collapse, rescuing FA performance in deep convolutional networks.

What Problem Does This Paper Address?

In modern deep learning, backpropagation (BP) is the undisputed king. However, neuroscientists have long pointed out that BP is biologically implausible due to the "weight transport problem": it requires backward error-propagating synapses to share the exact same weights as forward synapses.

Feedback Alignment (FA) bypassed this issue by showing that networks can learn even if the backward pass uses fixed, random matrices. In shallow networks, the forward weights naturally "align" themselves to these random backward projections. Unfortunately, FA completely falls apart in deeper architectures like convolutional neural networks (CNNs). This paper tackles a fundamental question: What exactly stops weight alignment from happening in deep networks?

Methods

The authors first diagnose the geometry of FA gradients by measuring their effective rank (a continuous approximation of dimensionality). They observe a clear pathology: as deep FA networks attempt to train, their gradient trajectories suffer from severe rank collapse.

To combat this low-dimensional bottleneck, they evaluate two distinct mechanisms designed to inject dimensionality back into the training process:

  1. The Muon Optimizer: Unlike standard SGD, Muon applies an orthogonalization step (via Singular Value Decomposition) to the momentum matrix, effectively flattening its singular value spectrum (setting all non-zero singular values to 1). This forces the optimizer to explore lesser-used parameter directions.
  2. Batch Normalization (BN): Applied to hidden layer activations, BN has been shown to promote activation orthogonality, preventing the internal representations from collapsing into low-dimensional subspaces.

Additionally, they experiment with adding a local decorrelation loss to minimize channel covariance and use the Freon optimizer to smoothly interpolate between standard SGD and Muon geometry.

Datasets

The approaches were evaluated on four standard computer vision benchmarks of increasing difficulty:

  • CIFAR-10 (used for initial diagnostic analysis and depth scaling tests)
  • STL-10
  • CIFAR-100
  • Tiny ImageNet (a 200-class subset of ImageNet downsampled to 64x64)

Key Insights and Ideas

  • Rank Collapse is the Core Obstacle: In deep networks trained with standard optimizers, FA gradients quickly become highly correlated across successive steps. The network gets stuck updating the same few directions, preventing deeper layers from ever achieving alignment.
Figure 1: FA suffers from low dimensional gradients. A Increasing the number of layers results in lower performance in FA networks. B-H Metrics for the 4 convolutional layer model. B The process of feedback alignment makes the FA network converge much later than BP networks. C Weight alignment measured by the cosine similarity of the forward and feedback weights. The upper layers align, but the bottom layers do not. D Gradient alignment measured by the cosine similarity of the gradient obtained using the feedback weight and the transpose of the forward weights for the feedback projections. Again, the upper layers align, but the bottom layers do not. E Rank of the gradient in the last convolutional layer. The FA networks have much lower dimensionality. F Effective rank of the Gram matrix of the gradients. Again, FA networks have much lower dimensionality. G Cosine similarity of the gradients at step t and t+1. FA networks have similar gradients, which explains the low-dimensional gradient trajectory. H Possibly due to the low-dimensional gradients, the feature dimensionality is much lower than for the BP models.
  • Pure Noise is Not the Answer: One might think adding random noise would increase dimensionality and fix the issue. However, the authors show that adding isotropic Gaussian noise actually degrades performance (see Figure 7 in the paper). The dimensionality expansion must be structurally aligned with the true learning signal.
  • Complementary Geometric Interventions: Muon alters the geometry of the parameter updates, while Batch Normalization alters the geometry of the activation space. Because they attack rank collapse from two different angles, combining them yields a powerful synergistic effect.

Key Results

The geometric interventions successfully prevented the typical performance degradation seen in FA as networks get deeper.

Orthogonalizing updates with Muon restores gradient dimensionality and improves deep FA performance.

When scaling up to deeper models like ResNet-18 and AlexNet, the results were stark (Table 1):

  • ResNet-18 on CIFAR-100: Baseline FA completely fails, achieving a dismal 1.4% accuracy. Introducing both Muon and Batch Normalization rescues the network, boosting accuracy to 46.1%.
  • ResNet-18 on STL-10: Baseline FA achieves 14.5%, which skyrockets to 64.2% when Muon and BN are combined.
  • AlexNet on Tiny ImageNet: Baseline FA yields 6.4%, which increases to 32.1% with the combined approach.
Adding Batch Normalization counteracts representation collapse and stabilizes FA training.

Limitations and Future Work

While this work provides a major stepping stone for alternative credit-assignment strategies, a few key limitations remain:

  • The Backprop Gap: Even with Muon and BN, a significant performance gap still exists between FA and standard backpropagation on difficult tasks (e.g., 46.1% vs 75.2% for ResNet-18 on CIFAR-100).
  • Biological Plausibility of the Fixes: Neither Muon (which requires global SVD operations) nor standard Batch Normalization (which relies on global batch statistics) are biologically plausible.

The authors suggest that future work should focus on finding local, biologically realistic approximations to these mechanisms—such as Oja's rule for local principal component extraction or divisive normalization in neural circuits—to achieve high-dimensional update dynamics in a truly brain-like fashion.

Summary updated

Abstract

Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation. Interestingly, when training a network with fixed random feedback weights to circumvent this issue, learning aligns the forward weights with the feedback weights, leading the backpropagated error signal to become an approximation of the standard gradient used by BP. This process, called Feedback Alignment (FA), occurs in MLPs and very shallow CNNs but does not scale well to deeper architectures. In this work, we first investigated differences between BP and FA models, trained on CIFAR10, specifically focusing on the effective rank of the signal. We found that the FA error has a considerably lower rank and hence is constrained to a lower-dimensional subspace compared to BP, limiting exploration of the parameter space. Motivated by this observation, we evaluated two mechanisms for increasing the effective dimensionality of FA: Muon, an optimiser that orthogonalises weight updates; and hidden activity normalisation, which promotes activation orthogonality. Across larger architectures and benchmarks, we find that these methods consistently improve over FA baselines, for example, on CIFAR100 with a Resnet-18, accuracy increases by 9 percentage points. Our results identify low-dimensional gradient dynamics as a key obstacle to scaling FA and suggest that inducing higher-dimensional update geometry is a promising route toward scaling alternatives to backpropagation.
BibTeX
@misc{boeshertz2026overcoming,
  author = {Gauthier Boeshertz and Razvan Pascanu and Claudia Clopath},
  title = {{Overcoming Rank Collapse in Feedback Alignment}},
  year = {2026},
  eprint = {2606.11123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.11123},
}

Loading related content...

Figure