Lazy Training Dynamics in Overparameterized Neural Networks

This direction is split into other directions: lazy training regime (lazy training stability). Its last version stays readable and citable here.

Version 1 of 188 papers117 conceptsfrozen 2026-10-04

In classical optimization, we expect parameters to move significantly to find a needle in a haystack—a global minimum hidden within a complex, non-convex landscape. However, as neural networks grow wider, a surprising phenomenon emerges: the parameters barely move at all. This regime, often called lazy training, allows massive networks to converge to global optima by behaving like simple linear models.

The Intuition of Weight Stability

The core mystery of deep learning is why simple gradient descent reliably finds global minima in non-convex landscapes. Research by Simon Shaolei Du and Simon S. Du suggests that "over-parameterization" is the key. When a network is sufficiently wide, it possesses an enormous amount of redundant capacity.

Think of it as a massive choir trying to hit a specific note. If the choir has only two singers, each must change their pitch significantly to correct an error. But if the choir has ten thousand singers, each individual only needs to adjust their volume or pitch by an infinitesimal amount for the collective sound to reach the target.

In Gradient Descent Provably Optimizes Over-parameterized Neural Networks, the authors prove that in very wide networks, individual weight vectors stay remarkably close to their random initializations. This stability ensures that the activation patterns of neurons—which usually shift wildly during training—remain almost constant. As Mahdi Soltanolkotabi and Samet Oymak describe it, gradient descent effectively takes the "shortest path" from initialization to a global optimum, bypassing the chaotic non-convexity of the broader landscape.

The Neural Tangent Kernel

The mathematical formalization of this "laziness" is the Neural Tangent Kernel (NTK), introduced by Arthur Jacot, Clément Hongler, and Franck Gabriel in Neural Tangent Kernel: Convergence and Generalization in Neural Networks.

The NTK describes how the output of a network changes in response to updates in its parameters. In the infinite-width limit, the NTK becomes a static matrix that does not change during training. This means the network's evolution is governed by a fixed kernel, making it mathematically equivalent to a linear model (like Kernel Ridge Regression) using fixed features.

This linearization simplifies everything:

  1. Convergence: Optimization becomes a convex problem. Amit Daniely showed that SGD can learn a rich class of functions (the conjugate kernel class) in polynomial time because the weights stay within a local region that preserves these kernel properties.
  2. Acceleration: Momentum methods, which are well-understood in convex settings, can be proven to work here too. Jun-Kun Wang and Tao Wei demonstrated that Nesterov’s acceleration and Polyak’s momentum achieve their theoretical speedups in wide networks because the underlying "high-resolution" dynamical system remains stable.

Stability as a Functional Regularizer

Lazy training isn't just a theoretical curiosity; it explains why large models are more robust in practical scenarios like Federated Learning and Continual Learning.

In Federated Learning, "client drift" occurs when local models trained on different data distributions wander too far from each other, making aggregation difficult. Haolin Wang and Like Jian found that widening the network mitigates this. Because wider models operate in the lazy regime, local updates remain small and stay close to the global initialization, preventing the models from diverging into incompatible regions of the parameter space.

Similarly, in Continual Learning, the primary enemy is "catastrophic forgetting." Iman Mirzadeh and Arslan Chaudhry showed that wider networks forget less because their weights move so little. By staying near the initial state, the model avoids overwriting the specific parameter configurations learned for previous tasks. This proximity acts as a form of implicit functional regularization.

The Passive Bulk of Parameters

Recent empirical work has confirmed that this "laziness" is a dominant feature of modern training. Nora Belrose and Adam Scherlis analyzed the Training Jacobian and found that up to two-thirds of a network's parameters act as "passive passengers." These weights are essentially indistinguishable from random noise even after training, as noted by Matthias Thamm.

While this might seem inefficient, it is exactly what allows the network to remain stable. The "bulk" of the parameter space provides a high-dimensional buffer that ensures the Lipschitz constant of the network—its sensitivity to input perturbations—remains bounded and independent of width, as proven by Apostolos Evangelidis.

The Limits of Laziness

Despite its power, lazy training is an approximation. Matus Telgarsky pointed out a "tantalizing gap": while linearized NTK models can approximate any function, they may require exponentially more width than a standard "active" neural network to achieve the same precision.

The research suggests that while deep learning can be lazy, its true power might lie in the transition between this stable, linear-like regime and a more "active" regime where features are learned and weights move significantly. Understanding this boundary remains one of the most active frontiers in the field.

Papers in this version

Showing 60 of 88 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lazy_training_dynamics_in_overparameteri_810d9422,
  title = {Lazy Training Dynamics in Overparameterized Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lazy-training-dynamics-in-overparameterized-neural-networks/rdr_810d9422d1d745908cd2199f911e43f4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lazy_training_dynamics_in_overparameteri_810d9422_v1,
  title = {Lazy Training Dynamics in Overparameterized Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lazy-training-dynamics-in-overparameterized-neural-networks/rdr_810d9422d1d745908cd2199f911e43f4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}