Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?

Eduardo Cueto-Mendoza, John D. KelleherarXiv 2026 · arXiv

TL;DR

Grokking—the phenomenon where a model suddenly generalizes long after overfitting—might seem like a late-game win for performance, but it doesn't rescue training efficiency. This study demonstrates that because grokking occurs so late in the training process, the massive "energy debt" accumulated outweighs the sudden jump in accuracy, leaving the overall efficiency curve on a steady downward slide.

What Problem Does This Paper Address?

In deep learning, "grokking" is a fascinating and somewhat mysterious behavior where a model’s test accuracy remains at chance levels for thousands of epochs while training accuracy is near perfect, only to suddenly "click" and jump to high generalization.

The authors ask a practical question: Does grokking modify the dynamics of training efficiency? Usually, training efficiency (defined here as accuracy per unit of energy consumed) decreases over time because energy continues to be spent while accuracy gains eventually plateau. Researchers wondered if the sudden, sharp spike in test accuracy during a grokking event could create a local "bump" or reversal in this efficiency decline, potentially justifying extremely long training runs.

Methods

The researchers tracked training and test accuracy alongside energy consumption (GPU, CPU, and RAM) to calculate an efficiency metric: $$Eff = \frac{(Acc_{train} + Acc_{test})/2}{\sum Energy}$$

To study this, they employed several key approaches:

  • Orthograd: A specific algorithm used to induce grokking by injecting large constant values into the loss function, triggering "softmax collapse."
  • Variable Precision: They tested the models using 16-, 32-, and 64-bit floating-point precision to see if numerical stability influenced the interaction between grokking and efficiency.
  • Time Horizons: They compared "short-horizon" runs (up to 500 epochs) where grokking was unlikely to occur, against "long-horizon" runs (up to 20,000 epochs).
  • Formal Proof: Beyond empirical data, they provided a mathematical proof (verified with the LEAN4 theorem prover) establishing the conditions under which efficiency remains monotonic.

Datasets

The study utilized two canonical tasks known for exhibiting grokking behavior:

  1. Modular Addition (mod 113): A common algorithmic task where the model learns to add integers modulo 113.
  2. MNIST: The classic handwritten digit classification dataset, used here with a LeNet-style CNN and a Multilayer Perceptron (MLP).

Key Insights and Ideas

  • The "Energy Debt" Dominates: The most significant insight is that grokking is "too little, too late" for efficiency. By the time the model generalizes, the denominator of the efficiency equation (cumulative energy) is so large that even a massive jump in the numerator (accuracy) cannot cause the overall ratio to rise.
  • Efficiency Monotonicity is Robust: Across different architectures (MLP vs. CNN), tasks, and numerical precisions, the downward trend of efficiency remained remarkably stable.
  • Overfitting as a Requirement: The experiments in Section 3.2.1 provide empirical evidence that grokking cannot be "forced" early in the training process; a sustained period of overfitting appears to be a prerequisite.
  • Architectural Sensitivity: CNNs appeared more sensitive to the Orthograd algorithm than MLPs, with grokking events occurring closer to the point of "softmax collapse" in convolutional architectures.

Key Results

The empirical results showed that while grokking was successfully induced in long-horizon experiments, it failed to alter the efficiency trajectory.

Figure 3: Long-horizon training for an MLP on modular addition. Even as test accuracy (dashed line) jumps, the efficiency curve (solid orange/red line) continues its monotonic decline.
  • Short Horizon: In runs up to 500 epochs, applying Orthograd did not induce grokking, and efficiency decreased smoothly as expected (see Figures 1 and 2).
  • Long Horizon: In 20,000-epoch runs, grokking was clearly visible. For example, in the MLP 16-bit modular addition task, grokking occurred around epoch 12,329. However, the efficiency curve showed no local increase at this transition.
  • Mathematical Confirmation: The authors proved that for efficiency to increase, the proportional increase in accuracy must be larger than the proportional increase in cumulative energy. In late-stage training, the energy increase is almost always proportionally larger than the accuracy gain.
Figure 6: A zoomed-in view of the grokking transition for a LeNet CNN on MNIST. The sharp rise in accuracy is clearly visible, yet the efficiency remains unphased.

Limitations and Future Work

  • Hardware Specificity: The authors acknowledge that their efficiency proxy is dependent on specific hardware and implementation details. Different energy-sampling rates or hardware architectures might yield slightly different curves.
  • Task Scope: The study is limited to two specific datasets. Future work could explore if more complex, real-world tasks exhibit different efficiency-grokking dynamics.
  • Early Grokking: While this paper suggests grokking cannot occur early, the authors note that if a method were discovered to trigger generalization much sooner, the efficiency dynamics might change significantly. Exploring such "early grokking" remains an open research path.

Summary updated

Abstract

We study whether grokking, the delayed emergence of out-of-distribution generalization after apparent overfitting, alters training efficiency in deep neural networks, where efficiency is measured as accuracy per unit energy. Using two canonical settings, modular arithmetic and MNIST, we analyze two neural network models under 16-, 32-, and 64-bit precision and use Orthograd to test whether grokking produces noticeable deviations from the typically decreasing dynamics of training efficiency. We consider both short training runs of up to 500 epochs and long training runs of up to 20,000 epochs, and in both cases identify grokking as a sharp improvement in test accuracy after a sustained train-test gap. In the short-horizon experiments, applying Orthograd at progressively stronger levels during early training does not induce grokking, and efficiency remains smoothly decreasing. In the long-horizon experiments, Orthograd induces grokking, but the efficiency curves remain approximately monotonic and show no appreciable local increase at the grokking transition. These results suggest that, in our setting, grokking does not materially alter overall training-efficiency dynamics.
BibTeX
@misc{cuetomendoza2026deep,
  author = {Eduardo Cueto-Mendoza and John D. Kelleher},
  title = {{Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?}},
  year = {2026},
  eprint = {2611.18396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18396},
}
Figure