The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization

Ben Adlam, Jeffrey Pennington

Abstract: Modern deep learning models employ considerably more parameters than required to fit the training data. Whereas conventional statistical wisdom suggests such models should drastically overfit, in practice these models generalize remarkably well. An emerging paradigm for describing this unexpected behavior is in terms of a double descent curve, in which increasing a model's capacity causes its test error to first decrease, then increase to a maximum near the interpolation threshold, and then decrease again in the overparameterized regime. Recent efforts to explain this phenomenon theoretically have focused on simple settings, such as linear regression or kernel regression with unstructured random features, which we argue are too coarse to reveal important nuances of actual neural networks. We provide a precise high-dimensional asymptotic analysis of generalization under kernel regression with the Neural Tangent Kernel, which characterizes the behavior of wide neural networks optimized with gradient descent. Our results reveal that the test error has non-monotonic behavior deep in the overparameterized regime and can even exhibit additional peaks and descents when the number of parameters scales quadratically with the dataset size.

Introduction

In classical statistics, the bias-variance tradeoff suggests that adding too many parameters to a model leads to overfitting, resulting in a U-shaped error curve. However, modern deep learning challenges this intuition: models often possess far more parameters than data points yet generalize excellently. The current "double descent" paradigm explains this by noting that test error spikes at the interpolation threshold (where parameters $p$ equal samples $m$) before decreasing again in the overparameterized regime. This paper, presented at ICML 2020, argues that this picture is incomplete. By providing a precise, high-dimensional asymptotic analysis, the authors uncover a more complex, multi-scale landscape of generalization.

Methodology

The authors analyze Kernel Ridge Regression using the Neural Tangent Kernel (NTK) of a single-layer fully connected network. The NTK describes the evolution of wide neural networks during gradient descent in the "lazy training" regime. To handle the high-dimensional limits ($m, n_0, n_1 \to \infty$), the study employs:

  • Random Matrix Theory (RMT) and Free Probability: Used to calculate the spectral properties of the large, random matrices defining the NTK.
  • Gaussian Equivalents: Nonlinear activation functions are replaced with linear counterparts sharing the same first and second moments to make the mathematics tractable.
  • Linear Pencils: A technique from non-commutative algebra used to compute exact traces for the test error.

The theoretical results are validated using synthetic datasets consisting of standard Gaussian vectors and labels generated by a "teacher" neural network, alongside empirical simulations on finite-sized networks.

The Triple Descent Landscape

The core contribution of this work is the identification of three distinct regimes of parameterization, revealing a triple descent phenomenon. As shown in Figure 1, the error curve is characterized by transitions at different scales of parameterization.

Schematic of the triple descent curve showing the classical, abundant, and superabundant parameterization regimes.
  1. Classical ($p < m$): The standard underparameterized regime where error follows the traditional U-shape.
  2. Abundant ($m < p < m^2$): The first overparameterized regime where double descent is typically observed, with a peak at the linear scaling threshold $p = m$.
  3. Superabundant ($p > m^2$): A regime where the number of parameters scales quadratically with the number of samples. Here, a second peak in test error emerges near $p \approx m^2$, followed by a final descent.

Empirical Results and Centering

The authors provide exact formulas for asymptotic training and test error (Theorem 1). A key finding is that the non-monotonic behavior deep in the overparameterized regime is specifically attributed to the kernel's sensitivity to second-layer weights.

Test loss behavior across linear (p/m) and quadratic (p/m²) scaling for various activation functions and the effect of centering.

As illustrated in Figure 2, the choice of activation function (ReLU, Tanh, Linear) shifts the error curves, but the multi-peak structure remains consistent. Furthermore, "centering" the model—subtracting the initial function value—is shown to be crucial. In noiseless settings, centered models achieve a significantly faster learning rate, where test error decays as $m^{-2}$ rather than the standard $m^{-1}$.

Discussion and Limitations

While this work provides a rigorous foundation for multi-scale generalization, several limitations remain. The analysis is currently limited to single-hidden-layer fully connected networks; extending this to deeper architectures like CNNs or Transformers involves significantly more complex mathematics. Additionally, the reliance on Gaussian input data may not fully capture the low-dimensional structures present in real-world datasets like images. Finally, the NTK framework describes "lazy training," and future work is needed to determine how these phenomena manifest in regimes where significant feature learning occurs.

Summary updated

Abstract

Modern deep learning models employ considerably more parameters than required to fit the training data. Whereas conventional statistical wisdom suggests such models should drastically overfit, in practice these models generalize remarkably well. An emerging paradigm for describing this unexpected behavior is in terms of a \emph{double descent} curve, in which increasing a model's capacity causes its test error to first decrease, then increase to a maximum near the interpolation threshold, and then decrease again in the overparameterized regime. Recent efforts to explain this phenomenon theoretically have focused on simple settings, such as linear regression or kernel regression with unstructured random features, which we argue are too coarse to reveal important nuances of actual neural networks. We provide a precise high-dimensional asymptotic analysis of generalization under kernel regression with the Neural Tangent Kernel, which characterizes the behavior of wide neural networks optimized with gradient descent. Our results reveal that the test error has non-monotonic behavior deep in the overparameterized regime and can even exhibit additional peaks and descents when the number of parameters scales quadratically with the dataset size.
BibTeX
@inproceedings{adlam2020neural,
  author = {Ben Adlam and Jeffrey Pennington},
  title = {{The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=rYUZdxDVSlc},
}
Figure