Abstract
This paper investigates the generalization performance of overparameterized 3-layer Neural Tangent Kernel (NTK) models. While 2-layer NTK models are well-studied, they lack the multi-layer interactions characteristic of deep learning. We demonstrate that for a specific "learnable set" of ground-truth functions, the test error of an overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons in both hidden layers. Crucially, our analysis reveals that the test error descends faster with respect to the number of neurons in the second hidden layer (closer to the output) than the first. Furthermore, 3-layer models exhibit significantly greater robustness to initialization choices, such as bias, compared to their 2-layer counterparts.
Introduction
Modern deep learning often defies the classical bias-variance tradeoff: models with far more parameters than training samples can perfectly interpolate noise yet still generalize remarkably well to new data. To understand this phenomenon, researchers frequently employ the Neural Tangent Kernel (NTK) framework, which describes the behavior of wide neural networks during training. While 2-layer NTK models have provided initial insights, they cannot capture the hierarchical interactions of deeper architectures. This study extends the analysis to a 3-layer NTK model to determine how depth and layer-specific width influence generalization in the overfitted regime.
The 3-Layer NTK Architecture
The study focuses on a fully connected 3-layer network using ReLU activations. To isolate the structural benefits of depth, the authors analyze a scenario where only the middle-layer weights ($W_0$) are trained, while the bottom-layer ($V$) and top-layer ($w$) weights remain fixed at their random initial states.
By deriving high-probability upper bounds for the generalization error of the minimum $\ell_2$-norm solution, the researchers can quantify how the number of neurons in the first ($p_1$) and second ($p_2$) hidden layers affects performance as the training sample size ($n$) grows.
Key Insights: The "Second Layer" Advantage
A primary finding of this research is that the hidden layer closer to the output plays a more dominant role in reducing test error. The theoretical upper bound shows that the error decreases at a rate of $O(1/\sqrt{p_2})$ relative to the second layer's width, but only at $O(\sqrt[4]{\log p_1 / p_1})$ for the first layer.
Numerical simulations validate these bounds. As shown in the figures above, fixing $p_1$ and increasing $p_2$ (panel d) results in a more pronounced drop in Mean Squared Error (MSE) than fixing $p_2$ and increasing $p_1$ (panel c). This suggests that in deep architectures, increasing the width of layers closer to the output provides a more significant improvement in generalization performance.
Robustness and Learnability
Beyond layer scaling, the 3-layer NTK model proves to be more versatile than shallower models. In 2-layer networks, the absence of bias terms prevents the model from learning odd-degree polynomials. In contrast, the interaction between layers in a 3-layer network allows it to learn a much wider "learnable set," including all finite-degree polynomials, even without explicit bias.
Furthermore, 3-layer models are more stable in high-dimensional settings. While 2-layer models are highly sensitive to how bias is scaled relative to the input dimension $d$, 3-layer models maintain consistent performance across different initialization schemes. This suggests that depth inherently provides a form of regularization or robustness against suboptimal initialization.
Conclusion
This work provides a rigorous theoretical foundation for the benefits of depth in overparameterized models. By showing that the second hidden layer is more influential for generalization and that 3-layer models are more robust to initialization than 2-layer models, the authors offer a clearer picture of why deep networks succeed where shallow ones might fail. Future work may extend these findings beyond the "lazy training" NTK regime to active feature learning and more complex architectures like Convolutional Neural Networks.
Summary updated