To understand how deep neural networks generalize, machine learning theorists often look for regimes where the complex mathematics of deep learning becomes tractable. One of the most prominent frameworks for this is the Neural Tangent Kernel (NTK), which describes the behavior of networks in the infinite-width limit. In this limit, networks undergo what is known as lazy training.
The core intuition of lazy training is simple: when a network is extremely wide, each individual weight only needs to change by a microscopic amount to achieve a massive shift in the collective output. As a result, during optimization, the network's weights stay remarkably close to their random initial values. Mathematically, this linearizes the network's dynamics, turning a highly non-linear deep network into a simple, fixed kernel method.
In SGD Learns the Conjugate Kernel Class of the Network, Amit Daniely leveraged this perspective to provide polynomial-time convergence guarantees for deep architectures. Similarly, Sanjeev Arora et al. utilized this regime in Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks to explain how overparameterized networks can quickly minimize training error. This linear approximation allows researchers to analyze complex optimization phenomena, such as early stopping under label noise (Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks by Mahdi Soltanolkotabi et al.) or the information-theoretic properties of infinite ensembles (Information in Infinite Ensembles of Infinitely-Wide Neural Networks by Ravid Shwartz-Ziv et al.).
The Feature Learning Gap
However, lazy training comes with a major conceptual catch: the network does not learn new features. Because the internal representations remain essentially static throughout optimization, the network acts as a fixed random feature extractor, merely optimizing the final linear layer.
This creates a profound "feature learning gap" between theoretical models and practical deep learning. In reality, the power of deep networks lies in their ability to adaptively discover hierarchical representations tailored to the underlying data structure. As Peter L. Bartlett et al. survey in Deep learning: a statistical viewpoint, linear or lazy approximations fail to capture this non-linear feature adaptation, leaving a significant gap in our understanding of generalization.
Furthermore, the assumption that the kernel remains static does not hold for real-world architectures. For instance, Boris Hanin et al. demonstrated in Finite Depth and Width Corrections to the Neural Tangent Kernel that for networks with finite width and depth, the NTK is stochastic at initialization and evolves significantly during training. Similarly, Colin Wei et al. showed in Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel that explicit regularization pushes networks away from their initialization, forcing them to learn informative features that fixed-kernel methods miss entirely.
Alternative Frameworks: Mean-Field and $\mu$P
To capture true representation learning, researchers have turned to alternative scaling frameworks that allow weights to move significantly, often called the rich or active feature learning regime.
Mean-Field Scaling
Instead of keeping weights close to initialization, mean-field scaling treats neurons as a collective system of interacting particles that evolve dynamically. In On feature learning in neural networks with global convergence guarantees, Zhengdao Chen et al. provided global convergence guarantees within this regime, proving that networks can minimize loss while actively developing data-dependent features. This framework has been extended to complex settings like reinforcement learning in Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory, where Yufeng Zhang et al. demonstrated that temporal-difference and Q-learning can successfully optimize representations beyond the static NTK view.
Maximal Update Parametrization ($\mu$P)
Under standard parameterizations, increasing a network's width to infinity causes it to collapse into the lazy NTK regime. However, $\mu$P adjusts the learning rates and initialization scales across layers to ensure that feature learning remains active even in the infinite-width limit. In Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization, Zixiang Chen et al. proved that deep networks under $\mu$P achieve both rich feature learning and global convergence, showing that features evolve significantly while maintaining the linear independence required for optimization.
The Continuum Between Lazy and Rich Dynamics
Rather than viewing lazy and rich training as completely separate worlds, recent work frames them as endpoints of a continuous spectrum.
Sebastian Neumayer et al. introduced this perspective in On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks, using unbalanced optimal transport to show that the scale of initialization defines a continuous transition between the kernel and feature learning regimes. Similarly, Francis Williams et al. showed in Gradient Dynamics of Shallow Univariate ReLU Networks that the implicit bias of shallow ReLU networks can shift smoothly from smooth cubic splines (the kernel regime) to sharp, adaptive linear splines (the active regime) depending entirely on initialization and scaling choices.
This transition is heavily influenced by optimization hyperparameters. For instance, Zhichao Wang et al. analyzed the evolution of weight and kernel spectra in Spectral Evolution and Invariance in Linear-width Neural Networks, demonstrating how learning rates and adaptive methods drive a network from a static, lazy state into a dynamic state that extracts task-relevant features.
Why the Distinction Matters in Practice
Understanding whether a model operates in the lazy or rich regime is not just a theoretical exercise; it fundamentally changes how we design and scale practical algorithms.
- Meta-Learning and Fine-Tuning: In meta-learning, algorithms like MAML rely on adapting to new tasks quickly. In How Fine-Tuning Allows for Effective Meta-Learning, Kurtland Chua et al. demonstrated that while lazy approximations fail when tasks only approximately share a structure, full network fine-tuning succeeds because it allows the weights to escape initialization and discover new representations.
- Scaling Laws: When predicting how model performance scales with compute and data, static kernel theories often underestimate efficiency. In A Dynamical Model of Neural Scaling Laws, Blake Bordelon developed a dynamical model showing that active feature learning accelerates performance gains beyond what lazy training predicts, explaining why training for more steps is often highly compute-efficient.
- Optimizer Design: By understanding how gradients map to weight updates across different scaling limits, Jeremy Bernstein et al. introduced a framework called modular dualization in Modular Duality in Deep Learning. This toolkit unifies large-scale optimization trends like $\mu$P and Shampoo to prevent models from getting trapped in feature-limiting lazy states, setting training speed records in the process.
By moving beyond the mathematical convenience of fixed kernels, this line of research provides a more accurate, representation-focused foundation for modern deep learning.