On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model

Peizhong Ju, Xiaojun Lin, Ness B. Shroff

Abstract

This paper investigates the generalization performance of overparameterized 3-layer Neural Tangent Kernel (NTK) models. While 2-layer NTK models are well-studied, they lack the multi-layer interactions characteristic of deep learning. We demonstrate that for a specific "learnable set" of ground-truth functions, the test error of an overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons in both hidden layers. Crucially, our analysis reveals that the test error descends faster with respect to the number of neurons in the second hidden layer (closer to the output) than the first. Furthermore, 3-layer models exhibit significantly greater robustness to initialization choices, such as bias, compared to their 2-layer counterparts.

Introduction

Modern deep learning often defies the classical bias-variance tradeoff: models with far more parameters than training samples can perfectly interpolate noise yet still generalize remarkably well to new data. To understand this phenomenon, researchers frequently employ the Neural Tangent Kernel (NTK) framework, which describes the behavior of wide neural networks during training. While 2-layer NTK models have provided initial insights, they cannot capture the hierarchical interactions of deeper architectures. This study extends the analysis to a 3-layer NTK model to determine how depth and layer-specific width influence generalization in the overfitted regime.

The 3-Layer NTK Architecture

The study focuses on a fully connected 3-layer network using ReLU activations. To isolate the structural benefits of depth, the authors analyze a scenario where only the middle-layer weights ($W_0$) are trained, while the bottom-layer ($V$) and top-layer ($w$) weights remain fixed at their random initial states.

Architecture of the 3-layer neural network model analyzed in the study.

By deriving high-probability upper bounds for the generalization error of the minimum $\ell_2$-norm solution, the researchers can quantify how the number of neurons in the first ($p_1$) and second ($p_2$) hidden layers affects performance as the training sample size ($n$) grows.

Key Insights: The "Second Layer" Advantage

A primary finding of this research is that the hidden layer closer to the output plays a more dominant role in reducing test error. The theoretical upper bound shows that the error decreases at a rate of $O(1/\sqrt{p_2})$ relative to the second layer's width, but only at $O(\sqrt[4]{\log p_1 / p_1})$ for the first layer.

MSE vs. hidden layer widths $p_1$ and $p_2$ under various noise levels.

Numerical simulations validate these bounds. As shown in the figures above, fixing $p_1$ and increasing $p_2$ (panel d) results in a more pronounced drop in Mean Squared Error (MSE) than fixing $p_2$ and increasing $p_1$ (panel c). This suggests that in deep architectures, increasing the width of layers closer to the output provides a more significant improvement in generalization performance.

Robustness and Learnability

Beyond layer scaling, the 3-layer NTK model proves to be more versatile than shallower models. In 2-layer networks, the absence of bias terms prevents the model from learning odd-degree polynomials. In contrast, the interaction between layers in a 3-layer network allows it to learn a much wider "learnable set," including all finite-degree polynomials, even without explicit bias.

Furthermore, 3-layer models are more stable in high-dimensional settings. While 2-layer models are highly sensitive to how bias is scaled relative to the input dimension $d$, 3-layer models maintain consistent performance across different initialization schemes. This suggests that depth inherently provides a form of regularization or robustness against suboptimal initialization.

Conclusion

This work provides a rigorous theoretical foundation for the benefits of depth in overparameterized models. By showing that the second hidden layer is more influential for generalization and that 3-layer models are more robust to initialization than 2-layer models, the authors offer a clearer picture of why deep networks succeed where shallow ones might fail. Future work may extend these findings beyond the "lazy training" NTK regime to active feature learning and more complex architectures like Convolutional Neural Networks.

Summary updated

Abstract

In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.
BibTeX
@inproceedings{ju2022generalization,
  author = {Peizhong Ju and Xiaojun Lin and Ness B. Shroff},
  title = {{On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=iP6P309JlMg},
}
Figure