The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization

Ben Adlam, Jeffrey Pennington

TL;DR

This paper challenges the popular "double descent" paradigm of machine learning generalization. By analyzing the Neural Tangent Kernel (NTK) in high dimensions, the authors prove that test error can exhibit a third "hump"—a triple descent—revealing that generalization behavior is far more complex and multi-scaled than previously thought, especially when the number of parameters scales quadratically with the dataset size.

What Problem Does This Paper Address?

Classical statistics tells us that as a model gets more complex (i.e., has more parameters), it eventually starts to overfit, causing test error to go up in a familiar U-shaped curve. Recently, however, the deep learning community has embraced the "double descent" phenomenon: after that initial U-curve, if you keep adding parameters past the "interpolation threshold" (where parameters equal data points), the test error actually starts going down again.

However, most theoretical explanations for double descent rely on overly simplified models, like linear regression or "random features" where only the last layer is trained. These models don't capture the full richness of actual neural networks. The authors ask: Is double descent the whole story? Or are there other "scales" of overparameterization where generalization behavior changes again?

Methods

The authors use the Neural Tangent Kernel (NTK) to study wide, fully-connected, single-hidden-layer neural networks trained with gradient descent.

To get precise results, they employ high-dimensional asymptotic analysis. They look at the limit where the number of samples ($m$), input features ($n_0$), and hidden units ($n_1$) all go to infinity while keeping their ratios fixed.

Their mathematical toolkit includes:

  • Random Matrix Theory (RMT): To handle the spectra of large, complex weight matrices.
  • Free Probability & Linear Pencils: Advanced techniques used to "linearize" the non-linear dependencies in the NTK, allowing them to derive exact analytical expressions for the training and test error.

As a theoretical paper, the primary "dataset" is a synthetic setup designed for rigorous proof, utilizing standard Gaussian vectors for inputs and labels generated by a "teacher" neural network. The authors also run simulations on finite-sized networks to confirm that their asymptotic theory holds in practice.

Key Insights and Ideas

The most striking insight of the paper is that there isn't just one transition point ($p=m$). There is a second critical scale when the number of parameters $p$ is roughly the square of the number of samples ($p \approx m^2$).

Schematic of the triple descent risk curve showing classical, abundant, and superabundant regimes.

This reveals three regimes of parameterization:

  1. Classical: $p < m$. The standard underparameterized regime.
  2. Abundant: $m < p < m^2$. The regime where double descent usually "ends."
  3. Superabundant: $p > m^2$. A regime where the NTK reaches its "deterministic" limit.

Between the Abundant and Superabundant regimes, the test error can rise and fall again, creating a triple descent. This happens because the "first-layer" part of the NTK has a different rank structure than the "second-layer" part, and they "peak" at different scales.

Key Results

The paper provides exact formulas that give the asymptotic training and test error for NTK ridge regression, mathematically proving that test error can be non-monotonic deep in the overparameterized regime.

Test loss curves demonstrating peaks at linear and quadratic scaling, and the effect of centering.

Furthermore, the authors show that centering matters. "Centering" the network (subtracting the initial output) significantly improves generalization and can even eliminate some of the non-monotonic "humps" in the error curve.

In the noiseless case, they find exceptionally fast learning curves where the test error drops as $m^{-2}$ (the square of the dataset size), which is much faster than the typical $m^{-1}$ rate. Empirical tests show a near-perfect match between their theoretical "triple descent" curves and actual gradient descent training on finite networks.

Limitations and Future Work

  • Architecture: The study is limited to single-hidden-layer fully-connected networks. While the NTK exists for deeper architectures (CNNs, ResNets), the math becomes significantly more grueling.
  • Data Distribution: The results assume Gaussian input data. Real-world data (like images) has much more structure, which might change where these "humps" occur.
  • The NTK Limit: The NTK describes networks in the "infinite width" limit. While the authors show it works for finite networks, there is a known "feature learning" regime in deep learning that the NTK doesn't fully capture.

The authors suggest that future work should investigate whether these multi-scale phenomena exist in broader classes of architectures and more realistic datasets.

Summary updated

Abstract

Modern deep learning models employ considerably more parameters than required to fit the training data. Whereas conventional statistical wisdom suggests such models should drastically overfit, in p...
BibTeX
@inproceedings{adlam2020neural,
  author = {Ben Adlam and Jeffrey Pennington},
  title = {{The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=rYUZdxDVSlc},
}
Figure