Introduction
In deep learning theory, the Neural Tangent Kernel (NTK) is a foundational concept that describes how infinitely wide neural networks behave like kernel regression during training. A persistent question has been whether the NTK is uniquely powerful or simply a complex manifestation of standard kernels used in statistics for decades.
While previous research linked the Laplace kernel to the NTK of shallow (two-layer) networks, it remained unclear if adding layers—increasing network depth—expanded the network's fundamental capacity to represent functions. This paper proves a surprising theoretical equivalence: in the infinite-width limit, deep neural networks and the classic Laplace kernel represent the exact same set of functions.
Methodology: Singularity Analysis and RKHS
To compare these models, the authors analyze their Reproducing Kernel Hilbert Spaces (RKHS), which essentially define the "library" of functions a kernel can express. The analysis is restricted to inputs on a unit sphere ($\mathbb{S}^{d-1}$), a standard theoretical setup.
The "size" and "richness" of an RKHS are determined by the decay rate of the coefficients in the kernel's Maclaurin power series. Because calculating these coefficients directly for deep networks is practically impossible, the authors employ Singularity Analysis from the field of analytic combinatorics. By treating the kernels as complex functions and identifying their "dominant singularities" (points where the function blows up or becomes undefined), they can deduce the asymptotic decay rate of the coefficients.
Key Insights and Theoretical Results
- Depth Doesn't Change the Space: The most striking finding is that for any depth $k \ge 1$, the RKHS of the NTK is identical to that of the Laplace kernel ($\mathcal{H}_{Lap}(\mathbb{S}^{d-1}) = \mathcal{H}_{N_k}(\mathbb{S}^{d-1})$). While depth might alter the network's inductive bias (which functions are easier to learn), it does not change its expressive power (which functions are mathematically possible to represent).
- The $n^{-3/2}$ Rule: Both the Laplace kernel and the deep NTK have Maclaurin coefficients that decay at a rate of $O(n^{-3/2})$. This shared decay rate is the mathematical "DNA" that defines their identical function spaces.
- Smoothness vs. Expressivity: The paper also explores Exponential Power Kernels ($e^{-\|x-y\|^\gamma}$). The authors prove that "rougher" kernels (smaller $\gamma$) result in strictly larger RKHSs. Because the Laplace kernel ($\gamma=1$) is less smooth than the Gaussian kernel ($\gamma=2$), it can represent a much wider array of functions, including those that are less regular.
Numerical Confirmation
As a theoretical paper, the authors use numerical simulations rather than traditional machine learning datasets to verify their proofs. They calculate the Maclaurin coefficients for the Laplace kernel and NTKs of various depths ($N_1$ through $N_4$).
By scaling the coefficients by $n^{3/2}$, the plots demonstrate that the values converge to distinct horizontal asymptotes. This confirms the theoretical $O(n^{-3/2})$ asymptotic decay rate across all tested network depths.
The empirical decay of NTK coefficients matches the theoretical predictions with high precision. For instance, as shown in the table, the theoretical constant for a 2-layer NTK ($N_1$) is $\approx 0.253975$, and the numerical result at $n=100$ is $0.261069$.
Limitations and Future Work
- Restricted Domain: The equivalence proof is currently limited to inputs on the unit sphere. Whether this exact relationship holds in the entire Euclidean space ($\mathbb{R}^d$) remains an open question.
- Activation Specificity: The analysis focuses strictly on ReLU activations. Different functions (like Sigmoid or Tanh) might yield different RKHS properties.
- The "NTK Regime": This analysis applies to the "lazy training" (infinite-width) regime. It does not account for "feature learning," where a finite-width network's weights move significantly from initialization, potentially altering the effective kernel.
Future work may investigate how different activation functions influence the RKHS and compare these "non-smooth" kernels against deep networks in practical, real-world tasks.
Summary updated