Characterizing the Spectrum of the NTK via a Power Series Expansion

Michael Murray, Hui Jin, Benjamin Bowman, Guido Montúfar

Introduction

In the "infinite width limit," neural networks behave as linear models governed by the Neural Tangent Kernel (NTK). The spectrum—the set of eigenvalues—of this kernel is a critical determinant of training dynamics, specifically how fast a network converges and how well it generalizes to new data. Despite its importance, analyzing the NTK spectrum for deep networks with general activation functions has historically been difficult, often relying on simplifying high-dimensional assumptions. This paper introduces a novel mathematical framework to characterize the NTK spectrum for fixed input dimensions and general activation functions across any depth.

Theoretical Framework

The authors utilize a Power Series Expansion based on Hermite polynomials to represent the NTK as a sum of Hadamard (entry-wise) powers of the input data’s Gram matrix. A core technical contribution is a set of recurrence formulas that calculate the NTK coefficients at layer $l$ based on the coefficients of layer $l-1$ and the Hermite coefficients of the activation function. This expansion allows for bounding the "effective rank" of the NTK and performing asymptotic analysis on eigenvalue decay, revealing how models prioritize simple versus complex patterns.

Empirical Validation

The theoretical findings were validated using two distinct datasets: Caltech101, representing real-world, low-rank natural image data, and Isotropic Gaussian Data, which serves as a baseline for high-rank, "flat" spectra.

NTK spectra comparison across datasets and depths.

Key Insights

  • Spectral Inheritance: The NTK inherits its spectral structure from the input data. If the input data is approximately low-rank (as seen in the Caltech101 plots), the NTK will be as well.
  • The Dominant Outlier: Every NTK studied exhibits a massive "outlier" eigenvalue, caused by the constant term in the power series (related to mean activation and bias). This outlier becomes increasingly dominant as the network gets deeper.
  • Activation Tail Influence: The decay rate of NTK coefficients is dictated by the activation function's tail behavior. ReLU (growing tail) leads to slow polynomial decay, while Tanh (flat tail) leads to much faster exponential decay.
  • Impact of Depth: Increasing depth concentrates the kernel's power into the first few eigenvalues, effectively reducing the number of features the network can easily learn.

Key Results and Discussion

The authors proved that the effective rank of the NTK is upper-bounded by a constant multiple of the input data's effective rank, explaining why deep networks do not necessarily explode in complexity. Furthermore, they demonstrated that the infinite-width NTK can be approximated to high precision using only the first 50 terms of the power series.

While the current theory focuses on fully connected feedforward networks (FCNNs), empirical evidence suggests these trends hold for Convolutional Neural Networks (CNNs). Future work aims to formalize proofs for CNNs and Residual Networks (ResNets) and address the "curse of dimensionality" in kernel theory bounds.

Summary updated

BibTeX
@inproceedings{murray2023characterizing,
  author = {Michael Murray and Hui Jin and Benjamin Bowman and Guido Montúfar},
  title = {{Characterizing the Spectrum of the NTK via a Power Series Expansion}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Tvms8xrZHyR},
}
Figure