TL;DR
Grokking—the phenomenon where a model suddenly generalizes long after overfitting—might seem like a late-game win for performance, but it doesn't rescue training efficiency. This study demonstrates that because grokking occurs so late in the training process, the massive "energy debt" accumulated outweighs the sudden jump in accuracy, leaving the overall efficiency curve on a steady downward slide.
What Problem Does This Paper Address?
In deep learning, "grokking" is a fascinating and somewhat mysterious behavior where a model’s test accuracy remains at chance levels for thousands of epochs while training accuracy is near perfect, only to suddenly "click" and jump to high generalization.
The authors ask a practical question: Does grokking modify the dynamics of training efficiency? Usually, training efficiency (defined here as accuracy per unit of energy consumed) decreases over time because energy continues to be spent while accuracy gains eventually plateau. Researchers wondered if the sudden, sharp spike in test accuracy during a grokking event could create a local "bump" or reversal in this efficiency decline, potentially justifying extremely long training runs.
Methods
The researchers tracked training and test accuracy alongside energy consumption (GPU, CPU, and RAM) to calculate an efficiency metric: $$Eff = \frac{(Acc_{train} + Acc_{test})/2}{\sum Energy}$$
To study this, they employed several key approaches:
- Orthograd: A specific algorithm used to induce grokking by injecting large constant values into the loss function, triggering "softmax collapse."
- Variable Precision: They tested the models using 16-, 32-, and 64-bit floating-point precision to see if numerical stability influenced the interaction between grokking and efficiency.
- Time Horizons: They compared "short-horizon" runs (up to 500 epochs) where grokking was unlikely to occur, against "long-horizon" runs (up to 20,000 epochs).
- Formal Proof: Beyond empirical data, they provided a mathematical proof (verified with the LEAN4 theorem prover) establishing the conditions under which efficiency remains monotonic.
Datasets
The study utilized two canonical tasks known for exhibiting grokking behavior:
- Modular Addition (mod 113): A common algorithmic task where the model learns to add integers modulo 113.
- MNIST: The classic handwritten digit classification dataset, used here with a LeNet-style CNN and a Multilayer Perceptron (MLP).
Key Insights and Ideas
- The "Energy Debt" Dominates: The most significant insight is that grokking is "too little, too late" for efficiency. By the time the model generalizes, the denominator of the efficiency equation (cumulative energy) is so large that even a massive jump in the numerator (accuracy) cannot cause the overall ratio to rise.
- Efficiency Monotonicity is Robust: Across different architectures (MLP vs. CNN), tasks, and numerical precisions, the downward trend of efficiency remained remarkably stable.
- Overfitting as a Requirement: The experiments in Section 3.2.1 provide empirical evidence that grokking cannot be "forced" early in the training process; a sustained period of overfitting appears to be a prerequisite.
- Architectural Sensitivity: CNNs appeared more sensitive to the Orthograd algorithm than MLPs, with grokking events occurring closer to the point of "softmax collapse" in convolutional architectures.
Key Results
The empirical results showed that while grokking was successfully induced in long-horizon experiments, it failed to alter the efficiency trajectory.
- Short Horizon: In runs up to 500 epochs, applying Orthograd did not induce grokking, and efficiency decreased smoothly as expected (see Figures 1 and 2).
- Long Horizon: In 20,000-epoch runs, grokking was clearly visible. For example, in the MLP 16-bit modular addition task, grokking occurred around epoch 12,329. However, the efficiency curve showed no local increase at this transition.
- Mathematical Confirmation: The authors proved that for efficiency to increase, the proportional increase in accuracy must be larger than the proportional increase in cumulative energy. In late-stage training, the energy increase is almost always proportionally larger than the accuracy gain.
Limitations and Future Work
- Hardware Specificity: The authors acknowledge that their efficiency proxy is dependent on specific hardware and implementation details. Different energy-sampling rates or hardware architectures might yield slightly different curves.
- Task Scope: The study is limited to two specific datasets. Future work could explore if more complex, real-world tasks exhibit different efficiency-grokking dynamics.
- Early Grokking: While this paper suggests grokking cannot occur early, the authors note that if a method were discovered to trigger generalization much sooner, the efficiency dynamics might change significantly. Exploring such "early grokking" remains an open research path.
Summary updated