Grokked Models are Better Unlearners

Yuanbang Liang, Yang Li

Grokking—the phenomenon where a model suddenly achieves strong generalization long after overfitting—is typically studied in the context of model performance. However, recent research reveals that grokking is also a "superpower" for machine unlearning. Grokked models are significantly more efficient at forgetting specific data, suffer less "collateral damage" to their remaining knowledge, and develop modular internal representations that allow for surgical data removal.

The Challenge of Machine Unlearning

As privacy regulations like GDPR and CCPA grant users the "right to be forgotten," machine unlearning has become a critical technical challenge. The goal is to remove the influence of specific training data from a model without the massive computational cost of retraining from scratch.

The core difficulty is entanglement. In most deep learning models, knowledge is "monolithic"—the parameters used to remember a specific "forget" item are often the same ones used for "retain" items. Consequently, attempting to delete one often breaks the other, leading to a sharp drop in overall accuracy. This research investigates whether the training regime itself—specifically the transition into "grokking"—can produce models that are inherently easier to unlearn.

Methodology

The researchers compared the effectiveness of standard unlearning algorithms—including Gradient Ascent (GA), Fisher Forgetting, and SCRUB—when applied to two different types of model checkpoints:

  1. Pre-grokking: Models stopped early, representing conventional training.
  2. Post-grokking: Models trained significantly longer until the "delayed generalization" transition occurred.
Figure 1: The grokking transition and its impact on accuracy metrics. The left panel shows the classic grokking divergence between training and test loss/accuracy. The right panel tracks how retain, unlearning, and testing accuracy evolve through this transition.

The study spans vision (CNNs and ResNets on CIFAR-10, SVHN, and ImageNet-100) and language (Phi-1.5 on the TOFU benchmark).

Why Grokking Enhances Unlearning

The paper identifies several structural reasons why grokked models are superior unlearners:

  • The Modularity Shift: Grokking is not just a drop in loss; it is a structural reorganization. The model shifts from a "dense memorizing" state to a "sparse generalizing" state. This creates modular "circuits" that are easier to manipulate independently.
  • Orthogonal Optimization Pathways: In pre-grokked models, gradients for "forget" and "retain" data are nearly identical (correlation $\approx 0.99$). In grokked models, these gradients become much more orthogonal (correlation $\approx 0.45$). This allows unlearning algorithms to "push" on the forget data without moving the retain data.
  • Landscape Flatness: Grokked models reside in flatter regions of the loss landscape. This inherent stability makes them more resilient to the parameter perturbations required during the unlearning process.

Key Results

The empirical results demonstrate a clear advantage for post-grokking models across multiple metrics:

  • Superior Performance: Grokked models achieved 6-8% better unlearning effectiveness while maintaining 10-20% higher accuracy on retained data compared to early-stopped versions.
  • Efficiency Gains: Post-grokking models reached target forget levels in 5-8 steps, whereas pre-grokking models required 15-20 steps—a 60-70% reduction in computational overhead.
Figure 2: Comparison of unlearning efficiency. Grokked models (blue) reach zero unlearning accuracy (UA) significantly faster than pre-grokked models (red) at both 15% and 50% forget rates.
  • Privacy and Robustness: Grokked models showed significantly better resistance to Membership Inference Attacks (MIA) and maintained their superior adversarial robustness even after the unlearning process.

Conclusion and Future Work

The "Grokking Tax"—the 5-6x longer training time required—remains a barrier to immediate practical use. However, the paper suggests that when you stop training is an orthogonal lever to how you unlearn. Future research may focus on inducing these beneficial properties (modularity and gradient orthogonality) through better initialization or optimizers like Sharpness-Aware Minimization (SAM), potentially bypassing the need for extended training.

Summary updated

Abstract

\emph{Grokking}—delayed generalization that emerges well after a model has fit the training data—has been linked to robustness and representation quality. We ask whether this training regime also helps with \emph{machine unlearning}, i.e., removing the influence of specified data without full retraining. We compare applying standard unlearning methods \textit{before} versus \textit{after} the grokking transition across vision (CNNs/ResNets on CIFAR, SVHN and ImageNet) and language (a transformer on a TOFU‑style setup). Starting from grokked checkpoints consistently yields (i) more \textbf{efficient forgetting} (fewer updates to reach a target forget level), (ii) \textbf{less collateral damage} (smaller drops on retained and test performance), and (iii) \textbf{more stable updates} across seeds, relative to early‑stopped counterparts under identical unlearning algorithms. Analyses of features and curvature further suggest that post‑grokking models learn \emph{more modular representations} with reduced gradient alignment between forget and retain subsets, which facilitates selective forgetting. Our results highlight \textbf{when} a model is trained (pre‑ vs. post‑grokking) as an orthogonal lever to \textbf{how} unlearning is performed, providing a practical recipe to improve existing unlearning methods without altering their algorithms.
BibTeX
@misc{liang2025grokked,
  author = {Yuanbang Liang and Yang Li},
  title = {{Grokked Models are Better Unlearners}},
  year = {2025},
  eprint = {2512.03437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.03437},
}
Figure