Introduction
Modern deep learning defies classical statistical wisdom: even when a model has far more parameters than data points (overparameterization) and perfectly fits noisy training data (overfitting), it often generalizes remarkably well to new data.
To understand this phenomenon, researchers frequently use the Neural Tangent Kernel (NTK), which approximates how neural networks behave during training in the infinite-width limit. While 2-layer NTK models are well-studied, they have significant "blind spots." For example, without a bias term, they struggle to learn odd-degree polynomials. This paper investigates 3-layer NTK models to determine if adding just one more hidden layer resolves these limitations and to understand how the interaction between layers affects the test error. The authors prove that 3-layer models are significantly more robust to architectural choices like bias and reveal a surprising asymmetry: increasing the number of neurons in the second hidden layer reduces test error much faster than doing so in the first.
Methodology: The 3-Layer Architecture
The authors analyze a fully-connected 3-layer neural network using ReLU activations. To keep the mathematics tractable while still capturing the complexity of layer interactions, they focus on a specific training regime: only the middle-layer weights are trained, while the bottom and top layers remain fixed at their initial random states.
Key technical steps in their analysis include:
- Defining the "Learnable Set": Characterizing a specific class of ground-truth functions ($F_{(3)}^{\ell_2}$) that the 3-layer NTK can theoretically recover.
- Deriving an Upper Bound: Providing a high-probability bound on the generalization error (the difference between the model's prediction and the true function).
- Kernel Composition: Demonstrating that the 3-layer kernel ($K^{Three}$) is essentially a composition of the kernels from the individual layers ($K^{RF}$ and $K^{Two}$).
Key Insights
The theoretical analysis yields several compelling insights into why depth helps generalization:
- The Power of the Second Layer (Layer Asymmetry): The paper reveals a fundamental "layer asymmetry." The test error decreases at a rate of $O(1/\sqrt{p_2})$ for the second hidden layer (closer to the output), but much slower at $O(\sqrt[4]{\log p_1 / p_1})$ for the first hidden layer (closer to the input). This suggests the second layer is more critical for governing generalization behavior.
- Bias Independence: A frustrating limitation of 2-layer NTK models is their sensitivity to the bias term; without it, they cannot learn certain functions. The authors prove that a 3-layer NTK (even without bias) can learn all finite-degree polynomials, making it a much more robust "universal" learner.
- High-Dimensional Stability: In high-dimensional settings (large $d$), 2-layer models often require finely-tuned bias settings to perform well. 3-layer models are far less sensitive to these choices, suggesting that depth inherently provides a "buffer" against poor architectural hyperparameters.
Numerical Validation and Results
Because this is a theoretical paper, the authors validate their mathematical theorems using synthetic datasets. Data points are sampled from a $d$-dimensional unit hyper-sphere, and ground-truth functions are specifically constructed polynomials designed to test the model's ability to learn both even and odd components.
The numerical simulations strongly confirm the theoretical bounds. As the number of neurons in the second layer ($p_2$) increases, the Mean Squared Error (MSE) drops significantly faster and reaches a lower floor compared to when only the first layer's width ($p_1$) is increased. Furthermore, they formally prove that the set of functions a 3-layer unbiased NTK can learn is strictly larger than that of a 2-layer unbiased NTK, and at least as large as a 2-layer biased NTK.
Limitations and Future Work
While this work provides a significant step forward, it has limitations. The analysis assumes only the middle layer is updated; a full analysis where all three layers are trained simultaneously remains an open challenge. Additionally, the proofs rely on the assumption that inputs and initial weights are uniformly distributed on a sphere, which rarely applies to real-world data like images. Future directions include extending this framework to even deeper networks (4+ layers) and exploring architectures like Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) within the NTK framework.
Summary updated