Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli, Florent KrzakalaarXiv 2026 · arXiv

TL;DR

This paper introduces BackLoFi, an extension of the "Neural LoFi" spectral theory that captures how later layers in a neural network influence the features learned by earlier ones. By deriving a local coupling between a neuron's incoming and outgoing weights, the authors provide a tractable spectral algorithm that recovers "hidden" features invisible to forward-only learning, significantly improving accuracy on tasks like CIFAR-10.

What Problem Does This Paper Address?

In deep learning, we know that backpropagation allows later layers to reshape earlier representations—a process called joint training. However, most spectral theories of feature learning (like the original Neural LoFi) are "greedy" or forward-only, meaning they build representations layer-by-layer without feedback.

The authors address a fundamental question: How does learning in later layers change which features earlier layers can discover? Without this feedback, a network might miss crucial low-level cues that only become relevant once a high-level detector (like an "eye" or "ear" detector) is formed.

Methods

The authors propose BackLoFi, a spectral surrogate for backpropagation. Key components include:

  • Backward Coupling: They derive a mathematical link between a neuron’s incoming weight row and its outgoing weight column. Essentially, the outgoing column acts as a "gain" that tells the incoming row which signals to amplify.
  • The BackLoFi Algorithm: Unlike standard SGD, this algorithm visits layer boundaries from the top down. It uses "upper-layer sensitivities" to generate sample-dependent effective labels for earlier layers.
  • Spectral Selection: Instead of iterative gradient steps, the method identifies features by computing the Singular Value Decomposition (SVD) of a cross-layer correlation operator ($A_\ell$). This operator correlates lower-layer features with upper-layer responses through the labels.
  • Paired Spikes: The theory shows that learning creates "rank-one spikes" in the weight matrices. These spikes are paired: the same intermediate neurons that learn to detect a feature also learn how to transmit it upward.
This diagram illustrates how the incoming row and outgoing column of a neuron $j$ are coupled. Backward feedback from upper layers changes the features learned below, which in turn refines how those signals are transmitted upward.

Datasets

The researchers evaluated their theory and algorithm using:

  • Binary CIFAR-10: A subset of the standard image dataset, specifically tasking the network to distinguish between animals and vehicles.
  • The "Deep Staircase" Model: A synthetic, hierarchical multi-index model designed by the authors. In this model, certain "hidden" features are mathematically invisible to a forward pass but can be "unlocked" by backward feedback.

Key Insights and Ideas

  • Unlocking the Invisible: The most striking insight is that backward feedback can make previously "invisible" features detectable. In the "Deep Staircase" experiment, forward-only learning fails to find certain directions because they don't correlate with the labels directly. BackLoFi finds them by using the "context" provided by the layers above.
  • The Rainbow Mechanism: The paper provides a dynamical explanation for "rainbow networks"—a phenomenon where trained networks exhibit linked low-rank covariance structures across layers. The authors show this emerges naturally from the shared neuron coefficients in the row-column coupling.
  • Function-Space Interpretation: Feature selection is framed as a variational problem: the network seeks a feature-response pair that maximizes correlation with the labels, subject to constraints on the "complexity" (kernel norms) of the features.
  • Efficiency of Feedback: Experiments show that passing back just a few "scores" (low-rank information) per sample is nearly as effective as using the full, exact backpropagation signal.

Key Results

  • Performance Gains: On binary CIFAR-10, BackLoFi significantly outperformed forward-only Neural LoFi. For example, a BackLoFi model trained on 5,000 images achieved accuracy comparable to a forward LoFi model trained on 50,000 images.
  • Approaching Gradient Descent: In both fully connected and convolutional architectures, BackLoFi's performance closely tracked that of standard Gradient Descent (GD), especially in data-scarce regimes.
  • Theoretical Validation: The authors proved that in three-layer networks, their discrete updates converge to a set of self-consistent continuous-time equations, providing a solid theoretical foundation for the observed "staircase" learning.

Limitations and Future Work

  • Small Initialization Regime: Much of the formal theory relies on a "small initialization" regime with smooth activations. Extending this to larger initializations or non-smooth activations (like standard ReLU without smoothing) remains a challenge.
  • Depth Scaling: While the paper provides a closed description for three-layer networks, extending the full continuous-time theory to very deep networks (e.g., ResNets or Transformers) is a logical next step.
  • Finite-Sample Guarantees: The authors note that establishing rigorous finite-sample guarantees for the implementation—rather than just the infinite-width limit—is an open area for research.

Summary updated

Abstract

During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.
BibTeX
@misc{dandi2026neural,
  author = {Yatin Dandi and Matteo Vilucchio and Hugo Tabanelli and Florent Krzakala},
  title = {{Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning}},
  year = {2026},
  eprint = {2610.32849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32849},
}
Figure