TL;DR
The theoretical framework of the Neural Tangent Kernel (NTK)—which assumes neural networks behave like linear models in the infinite-width limit—promises faster optimization, principled uncertainty estimates, and reduced catastrophic forgetting. This paper tests these claims empirically across realistic neural network architectures and discovers a sharp disconnect between theory and practice: real-world networks violate NTK assumptions, theory-derived algorithms perform poorly, and widening practical models does not prevent forgetting.
What Problem Does This Paper Address?
In recent years, the infinite-width limit has emerged as a dominant theoretical lens for studying overparametrized deep networks. In this asymptotic regime, networks enter "lazy training": weights barely move from their initialization, and training dynamics simplify to linear models governed by the NTK or Gaussian Processes.
However, convergence rates to this regime scale poorly with depth—often requiring layer widths polynomial ($m = \Omega(L^6)$) or exponential in depth $L$. Because realistic deep architectures (like WideResNets) are far deeper and narrower than theoretical bounds assume, the authors ask a fundamental question: Does infinite-width NTK theory actually describe the behavior of practical models, or does it offer false guidance for algorithm design?
Methods
The authors investigate three specific domains where NTK theory has inspired algorithmic claims:
- Second-Order Optimization: They test the conditions required for fast convergence of Natural Gradient Descent (NGD) under overparametrization. Specifically, they evaluate whether the empirical NTK Gram matrix maintains full rank ($\lambda_{\min}(G(0)) > 0$) and whether the network Jacobian remains stable ($C' \le 1/2$) around initialization using scalable, matrix-free automatic differentiation.
- Uncertainty Quantification (Neural Bandits): They evaluate
NeuralUCB, where exploration hyperparameters $\gamma_t$ are set using theoretical NTK regret bounds ($\tilde{\mathcal{O}}(\text{poly}(1/\sqrt{m}, L, t))$), comparing this against fixed exploration parameters and marginal likelihood calibration via the linearized Laplace approximation (LLA). - Continual Learning: They track catastrophic forgetting, task accuracy, and parameter displacement ($\|w_T - w_0\|_2 / \|w_0\|_2$) across task sequences, comparing toy shallow MLPs with practical WideResNets trained both under-converged (few epochs) and fully converged (50 epochs).
Datasets
The study evaluates these phenomena across synthetic benchmarks and standard real-world tasks (no new benchmarks are introduced):
- Synthetic Regression: A 2D toy problem ($N=16$) with noisy sinusoidal targets.
- Classification / Computer Vision: Subsets of CIFAR-10 ($N=400$ and $N=2000$) for optimization conditions; RotatedMNIST (9 tasks), SplitCIFAR-10 (5 tasks), SplitCIFAR-100 (20 tasks), and SplitTinyImageNet (20 tasks) for continual learning.
- Contextual Bandits: Six standard UCI classification benchmark datasets transformed into contextual bandits:
magic,avila,letter,covertype,bean, andpendigits.
Key Insights and Ideas
- Jacobian stability fails dramatically in practice: While the Gram matrix condition ($\lambda_{\min} > 0$) is easily satisfied even at modest widths, the Jacobian stability condition required for fast second-order convergence fails by many orders of magnitude across both shallow and deep networks.
- Theory-derived bandit exploration explodes: NTK-based confidence bounds grow astronomically large (e.g., $\gamma_t > 10^{30}$), causing bandit algorithms to spend all their time exploring. The algorithm degenerates into near-random action selection.
- Reduced forgetting in wide networks is an artifact of undertraining: While previous literature suggested wide networks forget less due to "lazy" orthogonal updates, the authors reveal this only happens when networks are undertrained. Widening an undertrained model lowers its peak accuracy, artificially shrinking the drop in performance. When networks are fully trained to convergence, wider architectures forget just as severely, and their weights continue to update substantially.
Key Results
- Optimization: For WideResNets and deep MLPs, the close-to-linearity metric $C'$ remained between $10^3$ and $10^{18}$, far exceeding the theoretical threshold of $0.5$. Adding more training data ($N=400 \to 2000$) worsened stability further.
- Contextual Bandits: Theory-driven
NeuralUCBproduced the second-worst cumulative regret across all UCI benchmarks, performing only slightly better than a purely random policy. In contrast, online evidence maximization via linearized Laplace matched the best grid-searched constant parameter without requiring prior knowledge of $\gamma$. - Continual Learning: For WideResNets on SplitCIFAR-100 and SplitTinyImageNet, relative parameter displacement $\|w_T - w_0\|_2 / \|w_0\|_2$ remained between $0.1$ and $0.4$ regardless of widening factor ($2^0$ to $2^5$). When trained to convergence, average forgetting did not decrease with network width.
Limitations and Future Work
- Scope of Architectures: The paper evaluates architectures commonly used today (MLPs, WideResNets). It leaves open whether novel, massively wider future architectures could exploit the kernel regime without sacrificing representation learning.
- Algorithm Performance: The authors emphasize that NTK-inspired methods are not universally uncompetitive, but rather that their NTK motivation is fundamentally flawed for practical settings.
- Moving Forward: The results serve as a cautionary tale: researchers should not rely on infinite-width lazy training assumptions to justify algorithm design. Instead, deep learning theory needs frameworks that incorporate feature learning and non-lazy regimes.
Summary updated