The Lazy Training Regime of Overparameterized Networks

49 concepts38 papers1 proposal

Deep neural networks are highly non-convex, meaning their loss landscapes should theoretically be riddled with local minima, saddle points, and chaotic regions. Yet, in practice, simple first-order optimization algorithms like Stochastic Gradient Descent (SGD) reliably find global minima with zero training error.

The answer to this paradox lies in a phenomenon known as the lazy training regime. When a neural network is made sufficiently wide (highly overparameterized), its optimization dynamics fundamentally simplify. Instead of carving out complex, highly non-linear feature representations, the network achieves global convergence by keeping its parameters remarkably close to their random initialization.

Weight Trajectory Localization

To understand why lazy training occurs, imagine trying to move a heavy object. If you have only one or two people, each person must exert massive effort and travel a great distance to move it. But if you have a million people, each person only needs to nudge the object by a fraction of a millimeter.

In an ultra-wide network, the learning task is distributed across an immense population of redundant hidden units. As Behnam Neyshabur et al. observe, individual hidden units undergo smaller weight updates relative to their random initialization as the total number of units increases. Li Wang et al. show that this relative displacement of weights scales inversely with the network width.

This phenomenon is called weight trajectory localization. Throughout the entire optimization process, the model weights remain within a tiny neighborhood of their initial random values (Spencer Frei et al.; Yuan Cao et al.). In fact, empirical analyses by Matthias Thamm et al. reveal that the vast majority of weights in a trained network remain statistically indistinguishable from random noise, confirming that optimization only modifies a small fraction of the available weight space. The network changes its global output function through the collective, coordinated effect of these microscopic adjustments (Matus Telgarsky et al.; Difan Zou et al.).

Linearization and Kernel Invariance

Because individual parameters barely move from their starting positions, the network effectively operates within a highly local region of the loss landscape. In this local neighborhood, the network's output can be accurately approximated by a first-order Taylor expansion around its initialization.

As a result, wide neural networks evolve during gradient descent as if they were linear models (Jaehoon Lee et al.). The complex, non-convex objective function behaves as if it were nearly linear near the starting point (Zeyuan Allen-Zhu et al.).

A crucial consequence of this linearization is that the network's underlying Gram matrix—or Neural Tangent Kernel (NTK)—remains nearly constant throughout training (Jason D. Lee et al.; Seyed Mohammad Eshtehardian et al.). When training with sufficiently small learning rates, the global distribution of eigenvalues in the weight and kernel matrices remains invariant from initialization to convergence (Zhichao Wang et al.). The network essentially functions as a fixed kernel machine, inheriting the predictable convergence properties of reproducing kernel Hilbert spaces (Atsushi Nitanda et al.).

Why Lazy Training Guarantees Convergence

This kernel stability explains why optimization succeeds. In standard non-convex optimization, gradients can vanish or explode as the model moves through chaotic regions of the parameter space. However, in the lazy training regime, the stability of the Gram matrix's least eigenvalue is preserved by the network's width (Simon Shaolei Du et al.).

Because the kernel does not degenerate, the optimization process retains a descent condition similar to strong convexity throughout the entire trajectory. This allows simple gradient descent to achieve global convergence at a linear rate (Simon Shaolei Du et al.; Jason D. Lee et al.). Amit Daniely demonstrated that by maintaining weight trajectories close to their random initialization, SGD can successfully train deep networks in polynomial time, learning any function within the network's conjugate kernel class.

Practical Implications: Stability as Regularization

While lazy training originated as a theoretical framework to prove convergence, its core mechanism—weight stability—has profound practical implications across various deep learning paradigms:

  • Continual Learning: In continual learning, networks often suffer from catastrophic forgetting when learning new tasks. Etash Kumar Guha et al. demonstrate that increasing network width acts as a form of functional regularization. Because weights require smaller updates from their initial values to solve a new task, the proximity to the initial state prevents the model from drifting significantly and overwriting previously learned task logic.
  • Federated Learning: Distributed training often suffers from "client drift" due to heterogeneous local data. Like Jian et al. and Haolin Wang et al. show that wider models mitigate this issue. The weight stability of the lazy regime prevents local models from wandering too far toward local optima, keeping the local tangent kernels approximately constant and maintaining alignment with the global objective.
  • Adversarial and Generative Models: In Generative Adversarial Networks (GANs), optimization involves a delicate min-max game that is notoriously unstable. Yogesh Balaji et al. show that over-parameterization stabilizes GAN training by keeping weights near initialization, which preserves the stability of the mapping's Jacobian and prevents the optimization from falling into non-convergent limit cycles.

The Representational Trade-off

If lazy training makes optimization so stable and predictable, why isn't it the ultimate solution for all deep learning? The answer lies in a fundamental trade-off between optimization stability and feature learning.

Because the network behaves like a fixed kernel machine, it does not perform significant feature adaptation (Zhichao Wang et al.). It relies entirely on the representations present at random initialization. As Matus Telgarsky et al. point out, this comes at a steep cost in representational efficiency: while standard networks can approximate complex functions efficiently by learning features, linearized lazy networks require exponentially larger widths to achieve the same universal approximation performance.

Thus, the lazy training regime reveals a beautiful truth about deep learning: over-parameterization can turn a chaotic non-convex problem into a well-behaved linear one, but true representation learning often requires stepping outside this safe, lazy boundary.

Key Foundational Work

SGD Learns the Conjugate Kernel Class of the Network (2017) — contributed 2 concept elements to this direction.
Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018) — contributed 1 concept elements to this direction.
Gradient descent GAN optimization is locally stable (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lazy_training_regime_lazy_training_stabi_cdbba317,
  title = {lazy training regime (lazy training stability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lazy-training-regime-lazy-training-stability/rdr_cdbba317941e472bbb5242aa4a1076e7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lazy_training_regime_lazy_training_stabi_cdbba317_v1,
  title = {lazy training regime (lazy training stability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lazy-training-regime-lazy-training-stability/rdr_cdbba317941e472bbb5242aa4a1076e7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake