When learning a new task, humans often generalize almost instantly, grasping the underlying rule from a handful of examples. Deep neural networks, by contrast, frequently exhibit a stark "generalization lag": they rapidly memorize the training data to achieve perfect training accuracy, yet their validation performance remains at chance level for thousands of epochs before suddenly jumping to perfection.
This phenomenon, widely known as grokking, challenges traditional statistical learning theory. Why does a model that has already completely overfit its data suddenly change its strategy and discover the true underlying rules? The answer lies beneath the flat surface of validation curves, driven by a hidden competition between two entirely different optimization timescales.
Hidden Progress Beneath Flat Curves
During the long plateau where validation loss appears completely stagnant, it is easy to assume the network has stopped learning. However, standard performance metrics fail to capture the internal structural shifts occurring within the model.
As demonstrated by Boaz Barak et al. in Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit, optimization algorithms like Stochastic Gradient Descent (SGD) do not simply stumble into generalization by chance. Instead, they make continuous, incremental "hidden progress" by gradually amplifying feature signals in the gradient long before any change becomes visible in external loss or accuracy metrics.
Mechanistically, this progress represents a quiet battle between two distinct internal configurations: a memorization circuit and a generalizing circuit. In Progress measures for grokking via mechanistic interpretability, Neel Nanda et al. reverse-engineered transformers training on algorithmic tasks. They discovered that the network actually learns both circuits concurrently. The model quickly constructs a messy, high-complexity memorization solution to satisfy the training objective. Simultaneously, it slowly builds an elegant, structured generalizing circuit (such as a Fourier transform for modular arithmetic).
The sudden jump in validation accuracy—the "grokking" moment—occurs when the generalizing circuit finally becomes strong enough to take over, or when regularization actively strips away the redundant memorization components.
A Tale of Two Optimization Timescales
To understand why generalization lags so far behind memorization, we must look at the optimization dynamics of over-parameterized models. The learning process is characterized by a fundamental separation of timescales.
In A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation, Etienne Boursier et al. prove that under small regularization (such as weight decay), training operates in two distinct phases:
- The Fast Phase (Interpolation): The network rushes to minimize training error. Because memorizing specific data points requires less structural coordination than discovering global rules, the model quickly finds a high-norm, complex solution that interpolates the training set, driving training loss to zero.
- The Slow Phase (Drift): Once the training loss hits zero, the optimization landscape changes. The model lands on what is known as the zero-loss manifold—a continuous region of parameter space where training error remains zero. Here, explicit or implicit regularization takes over. The parameters begin a slow, steady "drift" along this manifold to minimize the model's weight norm.
This post-memorization cleanup is a process of norm minimization on the zero-loss manifold, as characterized by Tiberiu Musat in The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold. As weight decay compresses the parameters, it systematically dismantles the heavy, inefficient memorization circuits. Because the generalizing circuit is mathematically more parsimonious, it survives the compression, forcing the network to transition from rote memorization to systematic rules.
The duration of this delay is highly predictable. As shown by Pascal Junior Tikeng Notsawo et al. in Grokking Beyond the Euclidean Norm of Model Parameters, the temporal gap between memorization and generalization is inversely proportional to the product of the learning rate and the regularization strength.
The Geometry of the Transition
This geometric shift can also be viewed through the lens of representation compression. In Explaining Grokking and Information Bottleneck through Neural Collapse Emergence, Keitaro Sakamoto and Issei Sato connect delayed generalization to "Neural Collapse"—a phenomenon where the internal representations of the same data class shrink toward a single, highly symmetric geometric point. Because this geometric collapse occurs on a much slower timescale than initial data fitting, the model experiences a long performance plateau before the representations compress enough to unlock out-of-distribution generalization.
[ Random Initialization ]
│
▼ (Fast Phase: Rapid fitting of training points)
[ Memorized Solution ] <-- Training Loss = 0, Validation Accuracy = 0
│
▼ (Slow Phase: Regularization / Norm Minimization / Drift)
[ Geometric Collapse / Flattening ]
│
▼ (Phase Transition)
[ Generalized Solution ] <-- Training Loss = 0, Validation Accuracy = 100%
When this threshold is finally reached, the internal structural reorganization can be sudden and dramatic. Noa Rubin et al. in Grokking as a First Order Phase Transition in Two Layer Networks model this macro-level jump as a first-order phase transition. The network's internal weights abruptly snap from generic, noisy configurations into specialized "droplets" of features that perfectly align with the task's algebraic structure.
Crucially, this transition alters the geometry of the loss landscape itself. By isolating the training dynamics, Ting Han et al. demonstrated in Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking that the onset of generalization coincides strictly with the model moving into a "flat" minimum. Memorized solutions inhabit sharp, fragile valleys that fail under slight perturbations, whereas generalized solutions occupy broad, flat basins that inherently robustify the model against unseen data.
Universality Beyond Deep Learning
Because early studies focused heavily on transformers and modular arithmetic, delayed generalization was initially thought to be an exotic quirk of complex neural architectures or symbolic datasets. However, recent evidence reveals that it is a universal property of mathematical optimization.
In Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity, Jack W. Miller and Thang D Bui demonstrated that grokking naturally emerges in simpler paradigms like linear regression and Gaussian Processes. When an over-parameterized learning system balances data fit against model parsimony, the optimization struggle inherently creates a temporal gap where error minimization precedes structural simplification.
This was mathematically formalized by Mingyue Xu et al. in To Grok Grokking: Provable Grokking in Ridge Regression, providing a rigorous proof that delayed generalization occurs even in standard ridge regression. The sudden jump in validation performance is often an artifact of how we evaluate models using hard accuracy thresholds rather than continuous loss metrics (Noam Itzhak Levi et al.).
Ultimately, delayed generalization is not a mystery of deep learning, but a fundamental feature of gradient-based optimization: a model will always choose the path of least resistance to fit its data quickly, only refining that messy solution into an elegant, generalizable rule when forced to by the slow, steady pressure of regularization.