On the Disconnect Between Theory and Practice of Overparametrized Neural Networks

Jonathan Wenger, Felix Dangel, Agustinus Kristiadi

Abstract

Neural Tangent Kernel (NTK) theory has emerged as a dominant framework for understanding the optimization and generalization of deep neural networks. By assuming an infinite-width limit, NTK theory suggests that training dynamics simplify into a linear regime, often referred to as "lazy training." This paper provides a critical reality check on these assumptions, demonstrating that practical architectures—even those considered "wide" by engineering standards—do not operate within the theoretical bounds required for NTK-based guarantees. Through empirical evaluations of optimization stability, uncertainty quantification, and continual learning, we show that applying NTK theory blindly to practical models often results in poor algorithmic performance and misleading conclusions.

Introduction

In recent years, the deep learning community has sought a rigorous theoretical foundation to explain why overparametrized networks generalize so well. The Neural Tangent Kernel (NTK) has become a cornerstone of this effort. It posits that as a network's width approaches infinity, its Jacobian remains constant during training, allowing the model to be treated as a linear predictor or a Gaussian Process. This theoretical simplification offers enticing benefits: guaranteed convergence for second-order optimization, reliable uncertainty estimation, and a potential solution to catastrophic forgetting.

However, a significant gap exists between an "infinitely wide" mathematical abstraction and the finite WideResNets or MLPs used in practice. This study investigates whether the architectural choices made by practitioners allow them to reap the benefits promised by NTK theory, or if the theory describes a regime that is fundamentally disconnected from real-world deep learning.

Methodology

To evaluate the validity of the NTK regime, the authors conducted experiments across three distinct domains:

  1. Optimization and Jacobian Stability: Theory suggests that for fast convergence using methods like Natural Gradient Descent (NGD), the network's Jacobian should remain stable. The authors utilized a metric, $C'$, to quantify the deviation from linearity. A value of $C' \leq 0.5$ is typically required for theoretical convergence guarantees.
  2. Uncertainty Quantification in Bandits: The study tested NeuralUCB, an algorithm for sequential decision-making. They compared the theoretical exploration parameter ($\gamma_t$) derived from NTK bounds against practical heuristics across six UCI Machine Learning Repository datasets.
  3. Continual Learning: The "lazy training" hypothesis suggests that wider networks should experience less catastrophic forgetting because their weights move less from initialization. The authors tested this by measuring weight displacement ($||\theta_T - \theta_0||$) and forgetting rates on benchmarks like RotatedMNIST and SplitCIFAR100.

The experiments compared "theory-regime" setups (specific initializations and scaling) against "practice-regime" setups (standard PyTorch initializations and WideResNet architectures).

Key Insights

The Stability Illusion

Even at extreme widths—up to $10^7$ parameters for shallow networks—practical models never reach the stable Jacobian regime. The non-linearity measure ($C'$) remained several orders of magnitude above the theoretical threshold of 0.5. This suggests that second-order optimization strategies built on NTK assumptions are effectively operating on a non-existent foundation when applied to standard architectures.

Theoretical Over-Exploration

In bandit experiments, using the exploration parameter derived from NTK theory resulted in disastrous performance, often barely outperforming random guessing. The theoretical uncertainty bounds are so loose that the model spends nearly all its time exploring the environment and almost no time exploiting its learned knowledge. Simple constant exploration factors or marginal likelihood tuning significantly outperformed the "optimal" theoretical settings.

Width and Catastrophic Forgetting

While toy models show reduced forgetting as width increases, this effect vanishes in practical WideResNets trained to high accuracy. In fact, training a wider network to full convergence can sometimes lead to more forgetting than a narrower counterpart. This occurs because the wider network reaches a higher peak performance on initial tasks, creating a larger margin for performance degradation when learning new tasks. The relative change in weights remained large regardless of width, contradicting the core premise of lazy training.

Discussion and Results

The results indicate that "lazy training" is a theoretical abstraction that rarely describes how high-performing, practical networks learn. In the optimization experiments, the Jacobian changed significantly throughout training, indicating that feature learning—rather than simple kernel regression—is the primary driver of performance. In the bandit tasks, NeuralUCB was the second-worst performing method across all datasets, highlighting that theoretical bounds can be too conservative to be useful in practice.

Conclusion

The findings suggest that the current reliance on infinite-width limits may be leading deep learning theory down a path that is increasingly irrelevant to practical engineering. While NTK theory provides a useful mathematical sandbox, it fails to account for the non-linear dynamics and feature learning that characterize modern AI success. Future research should prioritize the development of theories that bridge this gap, perhaps focusing on frameworks like $\mu$-parametrization or other models that explicitly account for the evolution of representations during training. To build more reliable and efficient models, we must move beyond the "lazy" assumption and embrace the complexity of finite-width, non-linear networks.

Summary updated

Abstract

The neural tangent kernel (NTK) has garnered significant attention as a theoretical framework for describing the behavior of large-scale neural networks. Kernel methods are theoretically well-understood and as a result enjoy algorithmic benefits, which can be demonstrated to hold in wide synthetic neural network architectures. These advantages include faster optimization, reliable uncertainty quantification and improved continual learning. However, current results quantifying the rate of convergence to the kernel regime suggest that exploiting these benefits requires architectures that are orders of magnitude wider than they are deep. This assumption raises concerns that architectures used in practice do not exhibit behaviors as predicted by the NTK. Here, we supplement previous work on the NTK by empirically investigating whether the limiting regime predicts practically relevant behavior of large-width architectures. Our results demonstrate that this is not the case across multiple domains. This observed disconnect between theory and practice further calls into question to what degree NTK theory should inform architectural and algorithmic choices.
BibTeX
@misc{wenger2023disconnect,
  author = {Jonathan Wenger and Felix Dangel and Agustinus Kristiadi},
  title = {{On the Disconnect Between Theory and Practice of Overparametrized Neural Networks}},
  year = {2023},
  eprint = {2310.00137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00137},
}
Figure