Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew M. Saxe, Surya Ganguli

Modern neural networks owe their success to their ability to extract task-relevant features from data—a process known as the "rich" learning regime. However, much of our theoretical understanding is rooted in the "lazy" regime, where networks behave like simple linear models. This work bridges the gap by deriving exact solutions for a minimal model, revealing how the balance of initialization between layers dictates the transition between these two regimes.

The Lazy vs. Rich Divide

In deep learning theory, training typically falls into one of two categories:

  1. The Lazy Regime: The network stays close to its initialization, behaving like a linear model with a fixed Neural Tangent Kernel (NTK). While mathematically tractable, it often fails to capture the complex feature extraction seen in practice.
  2. The Rich Regime: The network actively learns new representations. This is the "magic" of deep learning, but it is notoriously difficult to analyze.

While the overall scale of initialization ($\tau$) is known to influence this transition, the role of relative scale between layers—unbalanced initialization—has remained largely unexplored.

Methods: Exact Solutions and Hyperbolic Geometry

The researchers analyzed a two-layer, single-neuron linear network. By applying hyperbolic and spherical coordinate transformations, they solved the gradient flow equations exactly. A key discovery was the identification of a conserved quantity, $\delta$, representing the difference in scale between the first and second layers.

Learning trajectories are constrained to hyperboloids defined by the conserved quantity $\delta$.

This quantity $\delta$ forces learning trajectories onto specific geometric shapes, such as hyperboloids. These constraints dictate how the network moves through parameter space, either smoothing the path toward feature learning or creating bottlenecks that delay it.

Key Insights: The Power of Upstream Imbalance

The study distinguishes between upstream initialization (where earlier layers are smaller or learn faster) and downstream initialization (where later layers dominate).

A major finding is the divergence between linear and nonlinear networks. In linear models, a "balanced" initialization is optimal for rich learning. However, in nonlinear networks, upstream imbalance acts as a catalyst. It triggers a rapid "alignment phase" where internal representations shift to match data patterns with minimal movement in actual weight values.

Results: Accelerating Feature Learning

The theoretical map provided by these exact solutions acts as a "weather map" for training. By adjusting $\tau$ and $\delta$, researchers can predict whether a network will exhibit lazy, rich, or "delayed rich" behavior.

Phase portraits showing Test Loss and Kernel Distance over time as a function of initialization parameters $\tau$ and $\delta$.

Experimental validation across various tasks confirmed these insights:

  • Grokking Speedup: In Transformers, decreasing the variance of the embedding layer (upstream imbalance) significantly reduced the time to "grok" modular arithmetic.
  • Interpretability: Upstream initializations in CNNs promoted the emergence of smooth, Gabor-like edge detectors.
  • Sample Efficiency: Modifying layer scales reduced the data needed to learn hierarchical structures in the Random Hierarchy Model.

Conclusion

This research demonstrates that initialization is not just about picking a starting point; it's about selecting the geometric "track" the network will follow. By engineering the learning speed profile across layers, specifically through upstream unbalanced initializations, we can significantly enhance the efficiency and interpretability of feature learning in deep networks.

Summary updated

Abstract

While the impressive performance of modern neural networks is often attributed to their capacity to efficiently extract task-relevant features from data, the mechanisms underlying this rich feature learning regime remain elusive, with much of our theoretical understanding stemming from the opposing lazy regime. In this work, we derive exact solutions to a minimal model that transitions between lazy and rich learning, precisely elucidating how unbalanced layer-specific initialization variances and learning rates determine the degree of feature learning. Our analysis reveals that they conspire to influence the learning regime through a set of conserved quantities that constrain and modify the geometry of learning trajectories in parameter and function space. We extend our analysis to more complex linear models with multiple neurons, outputs, and layers and to shallow nonlinear networks with piecewise linear activation functions. In linear networks, rapid feature learning only occurs with balanced initializations, where all layers learn at similar speeds. While in nonlinear networks, unbalanced initializations that promote faster learning in earlier layers can accelerate rich learning. Through a series of experiments, we provide evidence that this unbalanced rich regime drives feature learning in deep finite-width networks, promotes interpretability of early layers in CNNs, reduces the sample complexity of learning hierarchical data, and decreases the time to grokking in modular arithmetic. Our theory motivates further exploration of unbalanced initializations to enhance efficient feature learning.
BibTeX
@inproceedings{kunin2024get,
  author = {Daniel Kunin and Allan Raventós and Clémentine Dominé and Feng Chen and David Klindt and Andrew M. Saxe and Surya Ganguli},
  title = {{Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=eNM94i7R3A},
}
Figure