Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes

Zhenfeng Tu, Santiago Aranguri, Arthur Jacot

TL;DR

This paper provides a unifying framework for understanding neural network training by bridging the gap between "lazy" (linear) and "active" (feature-learning) regimes. The authors introduce a "mixed regime" where a network behaves lazily for small singular values but becomes active for large ones, offering the "best of both worlds": the easy convergence of lazy training and the powerful low-rank bias of active training.

What Problem Does This Paper Address?

In deep learning theory, researchers typically study two extremes of training dynamics:

  1. The Lazy Regime (NTK): Occurs with very wide networks and large initializations. The weights barely move, the dynamics are linear, and the network essentially acts like a fixed kernel. It's easy to analyze but fails to perform "feature learning."
  2. The Active/Rich Regime: Occurs with smaller initializations. The network learns complex features and exhibits a "low-rank bias," meaning it tries to find simple, low-dimensional structures in data. However, these dynamics are notoriously difficult to analyze and can get stuck at "saddle points" if not initialized perfectly.

The gap between these two has been a major hurdle. We didn't have a single mathematical description that covered both, nor did we fully understand the transition between them. This paper aims to unify these regimes into a single, granular theory.

Methods

The authors focus on shallow linear networks ($A_\theta = W_2 W_1$) trained with Gradient Descent on a Mean Squared Error (MSE) loss. Their core contribution is a self-consistent dynamics formula (Equation 1 in the paper) that approximates the evolution of the learned matrix $A_\theta$ for any initialization scale.

The key technical move is analyzing the training at the level of individual singular values. They show that a threshold exists at $\sigma^2 w$ (where $\sigma^2$ is the initialization variance and $w$ is the network width).

  • Singular values below this threshold follow lazy, linear dynamics.
  • Singular values that grow above this threshold switch to active, balanced dynamics.

To validate this, they derive a "phase diagram" using scaling laws, mapping out how the network behaves as a function of its width and initialization variance relative to the input dimension $d$.

Datasets

As a theoretical study, the paper uses synthetic data rather than standard benchmarks like ImageNet. The primary task is Low-Rank Matrix Recovery:

  • Goal: Recover a hidden rank-$K$ matrix $A^*$ from noisy observations $A^* + E$.
  • Setup: The noise $E$ and the initial weights are drawn from Gaussian distributions.
  • Evaluation: The authors measure the MSE (test error) and the "stable rank" of the learned matrix to see if the network correctly identifies the underlying low-rank structure of $A^*$.

Key Insights and Ideas

  • The Mixed Regime is the "Best of Both Worlds": Purely active networks often fail because they start in a state that isn't "aligned" with the task. In the mixed regime, the network starts lazy, which allows it to safely align itself with the data. Once aligned, the important singular values cross the $\sigma^2 w$ threshold and "speed up," fitting the signal while ignoring the noise.
  • Granular Dynamics: A network isn't just "lazy" or "active" as a whole. At any given time, different parts of the network (different singular values) can be in different regimes.
  • Incremental Learning: As singular values cross the threshold, they converge rapidly one by one. This explains the "incremental learning" behavior often seen in deep learning, where the network fits the most important patterns first.
Figure 1: For both plots, we train either using gradient descent or the self-consistent dynamics from equation (1), with the scaling γσ2 = −1.85, γw = 2.25 which lies in the active regime. (Left panel): We plot train and test error for both dynamics. We observe that the train/test error for gradient descent is very close to the train/test error for the self-consistent dynamics. (Right panel): We plot with a solid line the singular values of Aθ(t) when running the self-consistent dynamics, and use a dashed line for the singular values from running gradient descent. In this experiment, RankA⋆= 5. We use different colors for the 5 largest singular values and the same color for the remaining singular values. We can see how the 5 largest singular values ‘speed up’ as they cross the σ2w threshold, allowing them to converge earlier than the rest. The minimal test error is achieved in the short period where the large singular values have converged but not the rest.

Key Results

  • Unifying Formula: The authors prove that their self-consistent dynamics formula (Equation 1) accurately tracks actual Gradient Descent training with high probability.
  • The Phase Diagram: They successfully map the transitions between lazy and active behavior. They show that the lazy regime only persists for very large widths ($\gamma_w > 2$), which explains why it is rarely observed in practical, finite-width applications.
  • Low-Rank Recovery: In the active/mixed regime, the network successfully recovers the rank of the target matrix $A^*$, whereas the lazy regime fails by over-fitting the noise $E$.
Phase diagrams showing test error and recovered rank as a function of scaling parameters.

Limitations and Future Work

  • Linearity: The study is restricted to linear networks. While linear networks are a standard "testing ground" for deep learning theory, they lack the non-linear activations (like ReLU) used in practice.
  • Shallow Architecture: The analysis focuses on two-layer (shallow) networks. Extending this to deep networks is a logical next step, though the authors note that their results for shallow nets already go further than previous mean-field limits by accounting for finite widths.
  • Generalization to Non-linearities: A major open question is whether non-linear networks feature similar "mixed" regimes and if this threshold-crossing behavior is the secret to why deep learning converges so well in practice.

Summary updated

Abstract

The training dynamics of linear networks are well studied in two distinct setups: the lazy regime and balanced/active regime, depending on the initialization and width of the network. We provide a surprisingly simple unifying formula for the evolution of the learned matrix that contains as special cases both lazy and balanced regimes but also a mixed regime in between the two. In the mixed regime, a part of the network is lazy while the other is balanced. More precisely the network is lazy along singular values that are below a certain threshold and balanced along those that are above the same threshold. At initialization, all singular values are lazy, allowing for the network to align itself with the task, so that later in time, when some of the singular value cross the threshold and become active they will converge rapidly (convergence in the balanced regime is notoriously difficult in the absence of alignment). The mixed regime is the `best of both worlds': it converges from any random initialization (in contrast to balanced dynamics which require special initialization), and has a low rank bias (absent in the lazy dynamics). This allows us to prove an almost complete phase diagram of training behavior as a function of the variance at initialization and the width, for a MSE training task.
BibTeX
@inproceedings{tu2024mixed,
  author = {Zhenfeng Tu and Santiago Aranguri and Arthur Jacot},
  title = {{Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=9zQl27mqWE},
}
Figure