On the Disconnect Between Theory and Practice of Overparametrized Neural Networks

Jonathan Wenger, Felix Dangel, Agustinus KristiadiSubmission to ICLR 2024 · Rejected

TL;DR

The theoretical framework of the Neural Tangent Kernel (NTK)—which assumes neural networks behave like linear models in the infinite-width limit—promises faster optimization, principled uncertainty estimates, and reduced catastrophic forgetting. This paper tests these claims empirically across realistic neural network architectures and discovers a sharp disconnect between theory and practice: real-world networks violate NTK assumptions, theory-derived algorithms perform poorly, and widening practical models does not prevent forgetting.

What Problem Does This Paper Address?

In recent years, the infinite-width limit has emerged as a dominant theoretical lens for studying overparametrized deep networks. In this asymptotic regime, networks enter "lazy training": weights barely move from their initialization, and training dynamics simplify to linear models governed by the NTK or Gaussian Processes.

However, convergence rates to this regime scale poorly with depth—often requiring layer widths polynomial ($m = \Omega(L^6)$) or exponential in depth $L$. Because realistic deep architectures (like WideResNets) are far deeper and narrower than theoretical bounds assume, the authors ask a fundamental question: Does infinite-width NTK theory actually describe the behavior of practical models, or does it offer false guidance for algorithm design?

Methods

The authors investigate three specific domains where NTK theory has inspired algorithmic claims:

  1. Second-Order Optimization: They test the conditions required for fast convergence of Natural Gradient Descent (NGD) under overparametrization. Specifically, they evaluate whether the empirical NTK Gram matrix maintains full rank ($\lambda_{\min}(G(0)) > 0$) and whether the network Jacobian remains stable ($C' \le 1/2$) around initialization using scalable, matrix-free automatic differentiation.
  2. Uncertainty Quantification (Neural Bandits): They evaluate NeuralUCB, where exploration hyperparameters $\gamma_t$ are set using theoretical NTK regret bounds ($\tilde{\mathcal{O}}(\text{poly}(1/\sqrt{m}, L, t))$), comparing this against fixed exploration parameters and marginal likelihood calibration via the linearized Laplace approximation (LLA).
  3. Continual Learning: They track catastrophic forgetting, task accuracy, and parameter displacement ($\|w_T - w_0\|_2 / \|w_0\|_2$) across task sequences, comparing toy shallow MLPs with practical WideResNets trained both under-converged (few epochs) and fully converged (50 epochs).

Datasets

The study evaluates these phenomena across synthetic benchmarks and standard real-world tasks (no new benchmarks are introduced):

  • Synthetic Regression: A 2D toy problem ($N=16$) with noisy sinusoidal targets.
  • Classification / Computer Vision: Subsets of CIFAR-10 ($N=400$ and $N=2000$) for optimization conditions; RotatedMNIST (9 tasks), SplitCIFAR-10 (5 tasks), SplitCIFAR-100 (20 tasks), and SplitTinyImageNet (20 tasks) for continual learning.
  • Contextual Bandits: Six standard UCI classification benchmark datasets transformed into contextual bandits: magic, avila, letter, covertype, bean, and pendigits.

Key Insights and Ideas

  • Jacobian stability fails dramatically in practice: While the Gram matrix condition ($\lambda_{\min} > 0$) is easily satisfied even at modest widths, the Jacobian stability condition required for fast second-order convergence fails by many orders of magnitude across both shallow and deep networks.
  • Theory-derived bandit exploration explodes: NTK-based confidence bounds grow astronomically large (e.g., $\gamma_t > 10^{30}$), causing bandit algorithms to spend all their time exploring. The algorithm degenerates into near-random action selection.
  • Reduced forgetting in wide networks is an artifact of undertraining: While previous literature suggested wide networks forget less due to "lazy" orthogonal updates, the authors reveal this only happens when networks are undertrained. Widening an undertrained model lowers its peak accuracy, artificially shrinking the drop in performance. When networks are fully trained to convergence, wider architectures forget just as severely, and their weights continue to update substantially.

Key Results

  • Optimization: For WideResNets and deep MLPs, the close-to-linearity metric $C'$ remained between $10^3$ and $10^{18}$, far exceeding the theoretical threshold of $0.5$. Adding more training data ($N=400 \to 2000$) worsened stability further.
  • Contextual Bandits: Theory-driven NeuralUCB produced the second-worst cumulative regret across all UCI benchmarks, performing only slightly better than a purely random policy. In contrast, online evidence maximization via linearized Laplace matched the best grid-searched constant parameter without requiring prior knowledge of $\gamma$.
  • Continual Learning: For WideResNets on SplitCIFAR-100 and SplitTinyImageNet, relative parameter displacement $\|w_T - w_0\|_2 / \|w_0\|_2$ remained between $0.1$ and $0.4$ regardless of widening factor ($2^0$ to $2^5$). When trained to convergence, average forgetting did not decrease with network width.

Limitations and Future Work

  • Scope of Architectures: The paper evaluates architectures commonly used today (MLPs, WideResNets). It leaves open whether novel, massively wider future architectures could exploit the kernel regime without sacrificing representation learning.
  • Algorithm Performance: The authors emphasize that NTK-inspired methods are not universally uncompetitive, but rather that their NTK motivation is fundamentally flawed for practical settings.
  • Moving Forward: The results serve as a cautionary tale: researchers should not rely on infinite-width lazy training assumptions to justify algorithm design. Instead, deep learning theory needs frameworks that incorporate feature learning and non-lazy regimes.

Summary updated

Abstract

The infinite-width limit of neural networks (NNs) has garnered significant attention as a theoretical framework for analyzing the behavior of large-scale, overparametrized networks. By approaching infinite width, NNs effectively converge to a linear model with features characterized by the neural tangent kernel (NTK). This establishes a connection between NNs and kernel methods, the latter of which are well understood. Based on this link, theoretical benefits and algorithmic improvements have been hypothesized and empirically demonstrated in synthetic architectures. These advantages include faster optimization, reliable uncertainty quantification and improved continual learning. However, current results quantifying the rate of convergence to the kernel regime suggest that exploiting these benefits requires architectures that are orders of magnitude wider than they are deep. This assumption raises concerns that practically relevant architectures do not exhibit behavior as predicted via the NTK. In this work, we empirically investigate whether the limiting regime either describes the behavior of large-width architectures used in practice or is informative for algorithmic improvements. Our empirical results demonstrate that this is not the case in optimization, uncertainty quantification or continual learning. This observed disconnect between theory and practice calls into question the practical relevance of the infinite-width limit.
BibTeX
@misc{wenger2023disconnect,
  author = {Jonathan Wenger and Felix Dangel and Agustinus Kristiadi},
  title = {{On the Disconnect Between Theory and Practice of Overparametrized Neural Networks}},
  year = {2023},
  eprint = {2310.00137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00137},
}

Loading related content...

Figure