On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model

Peizhong Ju, Xiaojun Lin, Ness B. Shroff

Introduction

Modern deep learning defies classical statistical wisdom: even when a model has far more parameters than data points (overparameterization) and perfectly fits noisy training data (overfitting), it often generalizes remarkably well to new data.

To understand this phenomenon, researchers frequently use the Neural Tangent Kernel (NTK), which approximates how neural networks behave during training in the infinite-width limit. While 2-layer NTK models are well-studied, they have significant "blind spots." For example, without a bias term, they struggle to learn odd-degree polynomials. This paper investigates 3-layer NTK models to determine if adding just one more hidden layer resolves these limitations and to understand how the interaction between layers affects the test error. The authors prove that 3-layer models are significantly more robust to architectural choices like bias and reveal a surprising asymmetry: increasing the number of neurons in the second hidden layer reduces test error much faster than doing so in the first.

Methodology: The 3-Layer Architecture

The authors analyze a fully-connected 3-layer neural network using ReLU activations. To keep the mathematics tractable while still capturing the complexity of layer interactions, they focus on a specific training regime: only the middle-layer weights are trained, while the bottom and top layers remain fixed at their initial random states.

3-Layer Network Architecture

Key technical steps in their analysis include:

  1. Defining the "Learnable Set": Characterizing a specific class of ground-truth functions ($F_{(3)}^{\ell_2}$) that the 3-layer NTK can theoretically recover.
  2. Deriving an Upper Bound: Providing a high-probability bound on the generalization error (the difference between the model's prediction and the true function).
  3. Kernel Composition: Demonstrating that the 3-layer kernel ($K^{Three}$) is essentially a composition of the kernels from the individual layers ($K^{RF}$ and $K^{Two}$).

Key Insights

The theoretical analysis yields several compelling insights into why depth helps generalization:

  • The Power of the Second Layer (Layer Asymmetry): The paper reveals a fundamental "layer asymmetry." The test error decreases at a rate of $O(1/\sqrt{p_2})$ for the second hidden layer (closer to the output), but much slower at $O(\sqrt[4]{\log p_1 / p_1})$ for the first hidden layer (closer to the input). This suggests the second layer is more critical for governing generalization behavior.
  • Bias Independence: A frustrating limitation of 2-layer NTK models is their sensitivity to the bias term; without it, they cannot learn certain functions. The authors prove that a 3-layer NTK (even without bias) can learn all finite-degree polynomials, making it a much more robust "universal" learner.
  • High-Dimensional Stability: In high-dimensional settings (large $d$), 2-layer models often require finely-tuned bias settings to perform well. 3-layer models are far less sensitive to these choices, suggesting that depth inherently provides a "buffer" against poor architectural hyperparameters.

Numerical Validation and Results

Because this is a theoretical paper, the authors validate their mathematical theorems using synthetic datasets. Data points are sampled from a $d$-dimensional unit hyper-sphere, and ground-truth functions are specifically constructed polynomials designed to test the model's ability to learn both even and odd components.

MSE vs Neuron Counts

The numerical simulations strongly confirm the theoretical bounds. As the number of neurons in the second layer ($p_2$) increases, the Mean Squared Error (MSE) drops significantly faster and reaches a lower floor compared to when only the first layer's width ($p_1$) is increased. Furthermore, they formally prove that the set of functions a 3-layer unbiased NTK can learn is strictly larger than that of a 2-layer unbiased NTK, and at least as large as a 2-layer biased NTK.

Limitations and Future Work

While this work provides a significant step forward, it has limitations. The analysis assumes only the middle layer is updated; a full analysis where all three layers are trained simultaneously remains an open challenge. Additionally, the proofs rely on the assumption that inputs and initial weights are uniformly distributed on a sphere, which rarely applies to real-world data like images. Future directions include extending this framework to even deeper networks (4+ layers) and exploring architectures like Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) within the NTK framework.

Summary updated

Abstract

In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.
BibTeX
@inproceedings{ju2022generalization,
  author = {Peizhong Ju and Xiaojun Lin and Ness B. Shroff},
  title = {{On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=iP6P309JlMg},
}
Figure