In the theoretical study of deep learning, the Neural Tangent Kernel (NTK) is often used as a mathematical map to describe how a network's outputs change as its weights are updated. For several years, the dominant view was the "lazy training" regime, which suggests that in infinitely wide networks, this map remains fixed throughout training. However, recent empirical and theoretical evidence suggests that the success of real-world deep learning stems from the opposite: a "rich regime" where the kernel evolves dynamically to learn features from data.
The Static Map Fallacy
The original NTK framework, as explored by Geiger et al., posits that wide neural networks behave like linear models with a static kernel. In this "lazy" regime, the internal representations (features) do not change significantly from their random initialization. While this simplifies the math, it fails to capture the fundamental mechanism of deep learning: representation learning.
Vyas et al. demonstrate that this static view cannot explain the scaling laws observed in practice. They show that the empirical tangent kernel of a real network evolves and improves its feature quality at a constant rate throughout the entire training process. This continuous adaptation allows finite-width networks to generalize significantly better than their static kernel counterparts.
Mechanisms of Kernel Adaptation
If the kernel isn't static, how exactly does it change? Research has identified several structured ways the NTK evolves to match the task at hand:
- Spectral Alignment: Kopitkov et al. found that the top eigenfunctions of the kernel actively shift to align with the target function. This "neural memory" concentrates the network's capacity on the most relevant features, accelerating convergence.
- Kernel Rotation and Twisting: In practical architectures, the kernel undergoes significant rotation. Mok et al. identify this as "kernel twisting," noting that static metrics measured only at initialization are unreliable because they ignore how the kernel will rotate once training begins.
- Structural Emergence: As training progresses, the kernel often evolves from a label-agnostic state into a block-diagonal structure. Seleznova et al. show that this transition is a signature of feature learning, where samples of the same class develop higher correlations within the kernel space.
The Role of Finite Width and Depth
The departure from the static regime is largely driven by the physical constraints of real networks. Hanin et al. proved that the NTK is stochastic at initialization and evolves significantly during training, with these fluctuations growing exponentially as the ratio of depth to width increases.
In these finite-width settings, the kernel is not just a passive observer but a dynamically coupled variable. Zavatone-Veth et al. explain that depth induces a "scale-averaging" effect, where the network's output becomes a data-dependent mixture of representations. This coupling allows the model to fix errors in lower-level features through a process Li et al. call "backward feature correction."
Why Evolution Matters for Performance
The dynamic nature of the kernel is what allows networks to solve complex, non-linear tasks that static methods cannot. Frei et al. show that gradient descent can "amplify" weak random correlations into strong features, a process that requires the feature map to evolve.
This evolution is even observed in specialized training scenarios:
- Regularization: Lewkowycz et al. reveal that $L_2$ regularization acts as a "speed dial" for training by controlling the rate of exponential kernel decay.
- Task Shifts: When a network encounters new data, its feature learning mechanism can "reactivate." Lanzillotta et al. tracked NTK velocity and alignment to show that task shifts trigger abrupt, structured changes in the representation.
- Grokking: The mysterious phenomenon of "grokking"—sudden generalization long after overfitting—is described by Rubin et al. as a phase transition where the kernel abruptly shifts from noise to specialized task-specific features.
By moving beyond the assumption of a fixed kernel, researchers are beginning to bridge the gap between elegant theory and the messy, high-performing reality of deep neural networks. As Kothapalli suggests, the future of the field lies in "data-aware" kernels that can account for how representations evolve and collapse in response to the training data.
Key Foundational Work
Dynamics of Deep Neural Networks and Neural Tangent Hierarchy (2020) — contributed 3 concept elements to this direction.
Finite Depth and Width Corrections to the Neural Tangent Kernel (2020) — contributed 2 concept elements to this direction.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel (2020) — contributed 2 concept elements to this direction.