Six Lectures on Linearized Neural Networks

Theodor Misiakiewicz, Andrea Montanari

TL;DR

This paper provides a rigorous, pedagogical overview of linearized neural networks, explaining how overparametrized non-convex networks can be accurately modeled as linear systems via the "lazy training" or Neural Tangent Kernel (NTK) regime. By analyzing four foundational linear frameworks, the authors characterize phenomena like benign overfitting and double descent, while explicitly mapping out where linear approximations fail and how fully non-linear "rich" regimes take over.

What Problem Does This Paper Address?

One of the greatest paradoxes in modern machine learning is why deep neural networks—despite being highly non-linear and non-convex—optimize so easily to near-global optimality using simple first-order methods like Gradient Descent.

A leading hypothesis is overparametrization: when a network has far more parameters ($p$) than training samples ($n$), the optimization landscape simplifies. In this "lazy training" regime, the network's weights barely move from their random initialization, allowing the model to be accurately approximated by its first-order Taylor expansion. This paper systematically reviews the mathematical foundations of this linear correspondence to clarify exactly what linearized models can teach us about generalization, and where they fundamentally fall short.

Figure 1.1

Methods

The paper is structured as a series of six theoretical lectures analyzing four distinct mathematical abstractions of linearized networks:

  1. Linear Regression with Feature Concentration: A simplified model where features are characterized entirely by their mean and covariance, demonstrating how noisy covariates act as an implicit regularizer.
  2. Kernel Ridge Regression (KRR): The infinite-width limit of neural networks, analyzed in high dimensions using spherical harmonics and Gegenbauer/Hermite polynomial expansions.
  3. Random Feature (RF) Models: A framework where only the second-layer weights are trained, clarifying the statistical transition from finite width to infinite width.
  4. Neural Tangent Features: The finite-width NTK model, where concentration inequalities (like the Matrix Bernstein inequality) are used to prove how closely finite networks track their infinite-width counterparts.

Finally, the authors contrast these linear regimes with the Mean Field scaling regime, which uses partial differential equations (McKean-Vlasov equations) to model fully non-linear feature learning.

Datasets

As a purely theoretical tutorial, the paper does not utilize real-world empirical datasets. Instead, it evaluates the mathematical models using rigorous synthetic data distributions, primarily:

  • Covariates uniformly distributed on the $d$-dimensional sphere ($x_i \sim \text{Unif}(S^{d-1}(\sqrt{d}))$).
  • Isotropic and anisotropic Gaussian feature distributions ($z_i \sim \mathcal{N}(0, \Sigma)$).
  • Synthetic target functions constructed from low-degree Hermite polynomials and single-neuron "ridge functions" ($f_*(x) = \phi(\langle w_*, x \rangle)$).

Key Insights and Ideas

  • Self-Induced Regularization: In overparametrized settings, even if explicit regularization is set to zero ($\lambda = 0^+$), the high-frequency, non-linear components of the activation function effectively act as random noise. This noise implicitly regularizes the model, explaining why overparametrized networks can perfectly interpolate noisy data without destroying generalization (benign overfitting).
  • The Staircase Phenomenon: In high dimensions ($d \to \infty$), linearized models learn target functions in a discrete, step-by-step manner determined by sample size. If $n \propto d^\ell$, the network can fit polynomial components of degree $\ell$ but remains completely blind to higher-degree variations.
  • The Failure on Ridge Functions: Linearized networks suffer from a severe "curse of linearity." They cannot efficiently learn a simple "ridge function" (a function depending on a single direction, like a single neuron) using a polynomial number of samples.

Key Results

  • Sharp Parameter Conditions: The authors show that the finite-width neural tangent model tightly tracks the infinite-width KRR limit as long as the total number of parameters satisfies $Nd \gg n$ (modulo logarithmic factors).
  • Resolution of the Scaling Paradox: The paper formalizes how changing the network's output scaling factor from $\alpha_N \sim 1/\sqrt{N}$ (lazy/NTK regime) to $\alpha_N \sim 1/N$ (mean-field regime) completely alters the optimization dynamics, shifting the model from a simple linear system to a powerful, fully non-linear feature selector capable of learning ridge functions with only $O(d)$ samples.

Limitations and Future Work

The primary limitation highlighted is that linearized theory completely misses feature learning—the very mechanism that makes deep learning uniquely powerful in practice. Under lazy training, the network cannot adapt its internal representations to align with latent structures in the data.

The authors note that while the mean-field framework successfully captures this rich regime, tracking the evolution of weight distributions across deep, multi-layer architectures remains an open and mathematically formidable challenge for future research.

Summary updated

Abstract

In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. We first recall the correspondence between neural networks and linear models via the so-called lazy regime. We then review four models for linearized neural networks: linear regression with concentrated features, kernel ridge regression, random feature model and neural tangent model. Finally, we highlight the limitations of the linear theory and discuss how other approaches can overcome them.
BibTeX
@article{misiakiewicz2024six,
  author = {Theodor Misiakiewicz and Andrea Montanari},
  title = {{Six Lectures on Linearized Neural Networks}},
  journal = {Journal of Statistical Mechanics Theory and Experiment},
  year = {2024},
  doi = {10.1088/1742-5468/ad292a},
}
Figure