TL;DR
While training neural networks with large learning rates often triggers the "Edge of Stability" (EoS)—a regime where the network's sharpness oscillates rather than diverges—this paper reveals a hidden benefit: it forces the eigenvectors of the Neural Tangent Kernel (NTK) to align more strongly with the training target. This phenomenon, termed "alignment shift," occurs precisely when sharpness drops during EoS oscillations, providing a clear mathematical link between large learning rates, kernel evolution, and better generalization.
What Problem Does This Paper Address?
In deep learning theory, the Neural Tangent Kernel (NTK) is a crucial tool used to analyze how neural networks learn. Under small learning rates, the NTK remains virtually unchanged (the "lazy regime"). However, practical training uses large learning rates, pushing the network into the "rich regime" where the NTK evolves, and a phenomenon called the Edge of Stability (EoS) emerges. At EoS, the network's sharpness (the maximum eigenvalue of the loss Hessian) climbs until it hits a critical threshold ($2/\eta$, where $\eta$ is the learning rate) and begins to oscillate.
While previous research has deeply explored how NTK eigenvalues behave during EoS, the behavior of its eigenvectors has remained a mystery. Do they just rotate randomly during these violent sharpness oscillations, or is there an underlying pattern? This paper addresses this gap by investigating how NTK eigenvectors rotate and align with the training target during EoS.
Methods
The authors tackle this problem using a combination of empirical tracking across diverse deep learning architectures and rigorous mathematical modeling:
- Metrics for Alignment: The paper tracks Kernel Target Alignment (KTA), which measures the overall alignment of the NTK matrix with the target vector, alongside a newly defined metric: Individual Eigenvector Target Alignment, which maps out how much each specific eigenvector contributes to that alignment.
- Theoretical Analysis: The authors prove the mechanism mathematically using a two-layer linear network trained on unwhitened data. They break down the training dynamics into four distinct phases of EoS to track exactly when and why the eigenvectors rotate.
- Central Flow Framework: To isolate the implicit effects of EoS, they leverage the "central flow" framework, which models gradient descent at EoS as a continuous gradient flow penalized by an implicit sharpness-reduction term.
Datasets
The empirical findings are validated across several standard setups:
- Synthetic Data: Unwhitened Gaussian data ($X \in \mathbb{R}^{d \times n}$) with a linear target function, used specifically for verifying the two-layer linear network theory.
- CIFAR-10: A subset of 2,000 images (randomly selected from two classes) used to evaluate non-linear architectures, including a 4-layer fully-connected network, VGG-11, ResNet-20, and a Vision Transformer (ViT).
- IMDB: A subset of 1,024 examples used to evaluate a language transformer (BERT-small) on a binary text classification task.
Key Insights and Ideas
- The "Alignment Shift": Larger learning rates don't just accelerate training; they fundamentally alter the NTK's geometry. They cause the "distribution" of target alignment to shift heavily toward the leading (top) eigenvectors of the NTK.
- Sharpness Drops Drive Feature Learning: The alignment doesn't happen smoothly. Instead, it occurs in sudden, stair-step jumps precisely during the phases of EoS where the network sharpness decreases (the downward slope of the oscillation).
- The Power of $\eta^2$: Traditional gradient flow analyses often ignore higher-order learning rate terms ($\eta^2$). This paper demonstrates that the $\eta^2$ term is mathematically responsible for driving the increase in eigenvector alignment during the sharpness reduction phase.
- Implicit Sharpness Penalty: By comparing standard gradient flow to central flow, the authors show that the implicit sharpness penalty present at EoS is the core mechanism forcing the kernel to align with the target.
Key Results
- Higher KTA with Larger Learning Rates: Across all architectures, larger learning rates consistently led to higher final KTA values. For example, in a fully-connected network, a learning rate of 0.2 achieved significantly higher target alignment than a learning rate of 0.05.
- Asymmetric KTA Velocity: Quantitative tracking showed that the average increase in KTA per step is orders of magnitude larger during sharpness-decreasing periods than during sharpness-increasing periods (e.g., $0.0029$ vs $0.0003$ in the fully-connected network).
- Generalization Benefits: The authors empirically demonstrated that this alignment shift correlates with a sudden decrease in the normalized inverse NTK metric ($y^T K^{-1} y$), which directly translates to tighter generalization bounds and lower test loss.
Limitations and Future Work
- Linear Theory vs. Non-linear Reality: While the empirical observations hold true for complex architectures like Vision Transformers and BERT, the rigorous mathematical proofs are primarily restricted to a two-layer linear network (with only a partial generalization to a two-layer ReLU network in the appendix).
- Future Directions: A major open challenge is extending this theoretical framework to multi-layer non-linear networks. Additionally, the authors note that explicitly connecting this "alignment shift" to specific downstream feature learning properties remains an exciting direction for future research.
Summary updated