Abstract
Neural Tangent Kernel (NTK) theory has emerged as a dominant framework for understanding the optimization and generalization of deep neural networks. By assuming an infinite-width limit, NTK theory suggests that training dynamics simplify into a linear regime, often referred to as "lazy training." This paper provides a critical reality check on these assumptions, demonstrating that practical architectures—even those considered "wide" by engineering standards—do not operate within the theoretical bounds required for NTK-based guarantees. Through empirical evaluations of optimization stability, uncertainty quantification, and continual learning, we show that applying NTK theory blindly to practical models often results in poor algorithmic performance and misleading conclusions.
Introduction
In recent years, the deep learning community has sought a rigorous theoretical foundation to explain why overparametrized networks generalize so well. The Neural Tangent Kernel (NTK) has become a cornerstone of this effort. It posits that as a network's width approaches infinity, its Jacobian remains constant during training, allowing the model to be treated as a linear predictor or a Gaussian Process. This theoretical simplification offers enticing benefits: guaranteed convergence for second-order optimization, reliable uncertainty estimation, and a potential solution to catastrophic forgetting.
However, a significant gap exists between an "infinitely wide" mathematical abstraction and the finite WideResNets or MLPs used in practice. This study investigates whether the architectural choices made by practitioners allow them to reap the benefits promised by NTK theory, or if the theory describes a regime that is fundamentally disconnected from real-world deep learning.
Methodology
To evaluate the validity of the NTK regime, the authors conducted experiments across three distinct domains:
- Optimization and Jacobian Stability: Theory suggests that for fast convergence using methods like Natural Gradient Descent (NGD), the network's Jacobian should remain stable. The authors utilized a metric, $C'$, to quantify the deviation from linearity. A value of $C' \leq 0.5$ is typically required for theoretical convergence guarantees.
- Uncertainty Quantification in Bandits: The study tested NeuralUCB, an algorithm for sequential decision-making. They compared the theoretical exploration parameter ($\gamma_t$) derived from NTK bounds against practical heuristics across six UCI Machine Learning Repository datasets.
- Continual Learning: The "lazy training" hypothesis suggests that wider networks should experience less catastrophic forgetting because their weights move less from initialization. The authors tested this by measuring weight displacement ($||\theta_T - \theta_0||$) and forgetting rates on benchmarks like RotatedMNIST and SplitCIFAR100.
The experiments compared "theory-regime" setups (specific initializations and scaling) against "practice-regime" setups (standard PyTorch initializations and WideResNet architectures).
Key Insights
The Stability Illusion
Even at extreme widths—up to $10^7$ parameters for shallow networks—practical models never reach the stable Jacobian regime. The non-linearity measure ($C'$) remained several orders of magnitude above the theoretical threshold of 0.5. This suggests that second-order optimization strategies built on NTK assumptions are effectively operating on a non-existent foundation when applied to standard architectures.
Theoretical Over-Exploration
In bandit experiments, using the exploration parameter derived from NTK theory resulted in disastrous performance, often barely outperforming random guessing. The theoretical uncertainty bounds are so loose that the model spends nearly all its time exploring the environment and almost no time exploiting its learned knowledge. Simple constant exploration factors or marginal likelihood tuning significantly outperformed the "optimal" theoretical settings.
Width and Catastrophic Forgetting
While toy models show reduced forgetting as width increases, this effect vanishes in practical WideResNets trained to high accuracy. In fact, training a wider network to full convergence can sometimes lead to more forgetting than a narrower counterpart. This occurs because the wider network reaches a higher peak performance on initial tasks, creating a larger margin for performance degradation when learning new tasks. The relative change in weights remained large regardless of width, contradicting the core premise of lazy training.
Discussion and Results
The results indicate that "lazy training" is a theoretical abstraction that rarely describes how high-performing, practical networks learn. In the optimization experiments, the Jacobian changed significantly throughout training, indicating that feature learning—rather than simple kernel regression—is the primary driver of performance. In the bandit tasks, NeuralUCB was the second-worst performing method across all datasets, highlighting that theoretical bounds can be too conservative to be useful in practice.
Conclusion
The findings suggest that the current reliance on infinite-width limits may be leading deep learning theory down a path that is increasingly irrelevant to practical engineering. While NTK theory provides a useful mathematical sandbox, it fails to account for the non-linear dynamics and feature learning that characterize modern AI success. Future research should prioritize the development of theories that bridge this gap, perhaps focusing on frameworks like $\mu$-parametrization or other models that explicitly account for the evolution of representations during training. To build more reliable and efficient models, we must move beyond the "lazy" assumption and embrace the complexity of finite-width, non-linear networks.
Summary updated