Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

Lin Chen, Sheng Xu

Introduction

In deep learning theory, the Neural Tangent Kernel (NTK) is a foundational concept that describes how infinitely wide neural networks behave like kernel regression during training. A persistent question has been whether the NTK is uniquely powerful or simply a complex manifestation of standard kernels used in statistics for decades.

While previous research linked the Laplace kernel to the NTK of shallow (two-layer) networks, it remained unclear if adding layers—increasing network depth—expanded the network's fundamental capacity to represent functions. This paper proves a surprising theoretical equivalence: in the infinite-width limit, deep neural networks and the classic Laplace kernel represent the exact same set of functions.

Methodology: Singularity Analysis and RKHS

To compare these models, the authors analyze their Reproducing Kernel Hilbert Spaces (RKHS), which essentially define the "library" of functions a kernel can express. The analysis is restricted to inputs on a unit sphere ($\mathbb{S}^{d-1}$), a standard theoretical setup.

The "size" and "richness" of an RKHS are determined by the decay rate of the coefficients in the kernel's Maclaurin power series. Because calculating these coefficients directly for deep networks is practically impossible, the authors employ Singularity Analysis from the field of analytic combinatorics. By treating the kernels as complex functions and identifying their "dominant singularities" (points where the function blows up or becomes undefined), they can deduce the asymptotic decay rate of the coefficients.

Key Insights and Theoretical Results

  • Depth Doesn't Change the Space: The most striking finding is that for any depth $k \ge 1$, the RKHS of the NTK is identical to that of the Laplace kernel ($\mathcal{H}_{Lap}(\mathbb{S}^{d-1}) = \mathcal{H}_{N_k}(\mathbb{S}^{d-1})$). While depth might alter the network's inductive bias (which functions are easier to learn), it does not change its expressive power (which functions are mathematically possible to represent).
  • The $n^{-3/2}$ Rule: Both the Laplace kernel and the deep NTK have Maclaurin coefficients that decay at a rate of $O(n^{-3/2})$. This shared decay rate is the mathematical "DNA" that defines their identical function spaces.
  • Smoothness vs. Expressivity: The paper also explores Exponential Power Kernels ($e^{-\|x-y\|^\gamma}$). The authors prove that "rougher" kernels (smaller $\gamma$) result in strictly larger RKHSs. Because the Laplace kernel ($\gamma=1$) is less smooth than the Gaussian kernel ($\gamma=2$), it can represent a much wider array of functions, including those that are less regular.

Numerical Confirmation

As a theoretical paper, the authors use numerical simulations rather than traditional machine learning datasets to verify their proofs. They calculate the Maclaurin coefficients for the Laplace kernel and NTKs of various depths ($N_1$ through $N_4$).

Convergence of scaled Maclaurin coefficients

By scaling the coefficients by $n^{3/2}$, the plots demonstrate that the values converge to distinct horizontal asymptotes. This confirms the theoretical $O(n^{-3/2})$ asymptotic decay rate across all tested network depths.

Numerical vs. Theoretical coefficient values

The empirical decay of NTK coefficients matches the theoretical predictions with high precision. For instance, as shown in the table, the theoretical constant for a 2-layer NTK ($N_1$) is $\approx 0.253975$, and the numerical result at $n=100$ is $0.261069$.

Limitations and Future Work

  • Restricted Domain: The equivalence proof is currently limited to inputs on the unit sphere. Whether this exact relationship holds in the entire Euclidean space ($\mathbb{R}^d$) remains an open question.
  • Activation Specificity: The analysis focuses strictly on ReLU activations. Different functions (like Sigmoid or Tanh) might yield different RKHS properties.
  • The "NTK Regime": This analysis applies to the "lazy training" (infinite-width) regime. It does not account for "feature learning," where a finite-width network's weights move significantly from initialization, potentially altering the effective kernel.

Future work may investigate how different activation functions influence the RKHS and compare these "non-smooth" kernels against deep networks in practical, real-world tasks.

Summary updated

Abstract

We prove that the reproducing kernel Hilbert spaces (RKHS) of a deep neural tangent kernel and the Laplace kernel include the same set of functions, when both kernels are restricted to the sphere $\mathbb{S}^{d-1}$. Additionally, we prove that the exponential power kernel with a smaller power (making the kernel less smooth) leads to a larger RKHS, when it is restricted to the sphere $\mathbb{S}^{d-1}$ and when it is defined on the entire $\mathbb{R}^d$.
BibTeX
@inproceedings{chen2021deep,
  author = {Lin Chen and Sheng Xu},
  title = {{Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=vK9WrZ0QYQ},
}
Figure