When training overparameterized neural networks, we often witness a puzzling phenomenon: a model completely memorizes the training data, plateaus at perfect training accuracy with poor test performance, and then—long after convergence—suddenly begins to generalize. This delayed generalization, frequently observed as "grokking" or "epoch-wise double descent," challenges classical statistical learning theory.
Recent research reveals that this delay is not random. Instead, it is governed by a predictable, quantitative clock: the time required for a model to transition from memorization to generalization scales inversely with the strength of its regularization.
The Two-Timescale Optimization Process
To understand why generalization is delayed, we must look at the geometry of the loss landscape. In overparameterized models, there is an entire manifold of parameter configurations that can achieve zero training error. Some of these configurations rely on complex, high-norm memorization; others rely on simple, low-norm features that generalize well.
Optimization on this landscape unfolds across two distinct timescales:
- The Fast Phase (Interpolation): Driven by the massive gradients of the training loss, the network rapidly rushes toward the nearest point on the zero-training-error manifold. Because high-norm memorization is often easier to find, the model quickly interpolates the training data and overfits.
- The Slow Phase (Drift): Once training loss reaches zero, its gradients vanish. However, explicit regularization (such as weight decay) remains active. This introduces a much gentler, steady force that drags the parameters along the zero-loss manifold toward simpler, lower-norm solutions.
In A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation, Etienne Boursier et al. prove that this second phase is a continuous, steady movement toward simpler solutions. The "suddenness" of the generalization jump is often an illusion created by thresholding metrics like accuracy. As shown by Noam Itzhak Levi et al. in linear estimators, the underlying parameter trajectory is smooth, but the test accuracy only ticks upward once the parameters cross a specific geometric threshold.
[Initialization]
│
▼ (Fast Phase: Driven by Training Loss Gradient)
[Overfitted Interpolation] <-- Training error hits 0%
│
▼ (Slow Phase: Driven by Regularization Drift)
[Generalized Solution] <-- Test error drops
The Regularization Speed Dial
Because the slow phase is driven entirely by regularization, the velocity of this drift is directly proportional to the regularization coefficient ($\lambda$). If you double the regularization strength, the model moves twice as fast along the solution manifold. Consequently, the time ($T$) required to reach peak generalization scales inversely with the regularization coefficient:
$$T \propto \frac{1}{\lambda}$$
Aitor Lewkowycz et al. first demonstrated this in On the training dynamics of deep networks with $L_2$ regularization, framing $L_2$ regularization as a "speed dial" for training dynamics. This inverse relationship allows practitioners to trade training time for final model quality: larger regularization values accelerate the transition to generalization, while smaller values allow the model to find cleaner solutions at the cost of significantly longer training runs.
This inverse scaling is a universal property of structural regularization. In Omnigrok: Grokking Beyond Algorithmic Data, Ziming Liu et al. show that weight decay acts as the primary driver moving parameters from a high-norm overfitted state to a lower-norm generalizing state. Expanding on this, Pascal Junior Tikeng Notsawo et al. demonstrate in Grokking Beyond the Euclidean Norm of Model Parameters that this clock applies to any regularization (such as sparsity or low-rank constraints) that aligns with the true data structure. They formalize the generalization delay as inversely proportional to the product of the learning rate and the regularization strength.
Feature Learning and Neural Collapse
What is actually happening to the network's internal representations during this slow drift?
One perspective focuses on multi-scale feature learning. As explored by Mohammad Pezeshki et al. in Multi-scale Feature Learning Dynamics: Insights for Double Descent, networks learn different features at different speeds. Fast-scale features are quickly captured but lead to early overfitting, while slow-scale features take much longer to emerge but ultimately drive generalization. Regularization acts as a control mechanism: low weight decay allows these slow-scale features to eventually develop and trigger a second descent in error, whereas excessive regularization can suppress them entirely.
A complementary perspective views this drift as the emergence of Neural Collapse—a geometric phenomenon where the network's internal representations of the same class shrink toward a single, highly symmetric point.
Overfitted State Neural Collapse (Generalized)
▲ ▲ ▲
│ │ │
┌─┴─┐ ┌─┴─┐ ┌──┴──┐
│X_1│ │X_2│ │X_1,2│
└───┘ └───┘ └─────┘
Class Representations Representations collapse
are scattered and noisy to a single clean prototype
In Explaining Grokking and Information Bottleneck through Neural Collapse Emergence, Keitaro Sakamoto et al. show that weight decay accelerates this geometric refinement. The time required for feature clustering is inversely proportional to regularization strength; weight decay forces the network to compress irrelevant data and collapse class representations, shifting the model from memorization to generalization.
Crucially, the exact mathematical implementation of this regularization matters. Jim Zhao et al. demonstrate in Optimizer Choice Matters For The Emergence of Neural Collapse that decoupling weight decay from the gradient update (as done in AdamW) prevents certain forms of neural collapse, whereas coupling it (as in standard Adam or SGD) promotes it, fundamentally altering the geometric properties of the learned feature space.
Controlling the Clock
Because the generalization delay is determined by a predictable interaction between initialization and regularization, we can actively manipulate the training clock.
In To Grok Grokking: Provable Grokking in Ridge Regression, Mingyue Xu et al. prove that the duration of the overfitted state is quantitatively determined by the ratio of the initialization scale to the weight decay coefficient. If you increase the initial weight variance or lower the regularization strength, you extend the distance the model must travel along the manifold, thereby lengthening the generalization delay.
Conversely, we can shorten the delay using two levers:
- Initialization Scale: Mohamad Amin Mohamadi et al. show in Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition that smaller initializations allow networks to exit the "lazy" kernel regime faster and enter the rich feature-learning regime sooner, reducing the temporal gap between training convergence and test generalization.
- Temporal Tuning: Since regularization primarily does its work after the fast phase or during early representation formation, timing is highly sensitive. Johan Björck et al. demonstrate in Understanding Decoupled and Early Weight Decay that applying weight decay exclusively during the early stages of training is often sufficient to keep weight norms small and effective learning rates high, capturing most of the generalization benefits before the slow drift even concludes.
Key Foundational Work
On the training dynamics of deep networks with $L_2$ regularization (2020) — contributed 2 concept elements to this direction.
Understanding Decoupled and Early Weight Decay (2021) — contributed 2 concept elements to this direction.