Grokking—the phenomenon where a model suddenly achieves strong generalization long after overfitting—is typically studied in the context of model performance. However, recent research reveals that grokking is also a "superpower" for machine unlearning. Grokked models are significantly more efficient at forgetting specific data, suffer less "collateral damage" to their remaining knowledge, and develop modular internal representations that allow for surgical data removal.
The Challenge of Machine Unlearning
As privacy regulations like GDPR and CCPA grant users the "right to be forgotten," machine unlearning has become a critical technical challenge. The goal is to remove the influence of specific training data from a model without the massive computational cost of retraining from scratch.
The core difficulty is entanglement. In most deep learning models, knowledge is "monolithic"—the parameters used to remember a specific "forget" item are often the same ones used for "retain" items. Consequently, attempting to delete one often breaks the other, leading to a sharp drop in overall accuracy. This research investigates whether the training regime itself—specifically the transition into "grokking"—can produce models that are inherently easier to unlearn.
Methodology
The researchers compared the effectiveness of standard unlearning algorithms—including Gradient Ascent (GA), Fisher Forgetting, and SCRUB—when applied to two different types of model checkpoints:
- Pre-grokking: Models stopped early, representing conventional training.
- Post-grokking: Models trained significantly longer until the "delayed generalization" transition occurred.
The study spans vision (CNNs and ResNets on CIFAR-10, SVHN, and ImageNet-100) and language (Phi-1.5 on the TOFU benchmark).
Why Grokking Enhances Unlearning
The paper identifies several structural reasons why grokked models are superior unlearners:
- The Modularity Shift: Grokking is not just a drop in loss; it is a structural reorganization. The model shifts from a "dense memorizing" state to a "sparse generalizing" state. This creates modular "circuits" that are easier to manipulate independently.
- Orthogonal Optimization Pathways: In pre-grokked models, gradients for "forget" and "retain" data are nearly identical (correlation $\approx 0.99$). In grokked models, these gradients become much more orthogonal (correlation $\approx 0.45$). This allows unlearning algorithms to "push" on the forget data without moving the retain data.
- Landscape Flatness: Grokked models reside in flatter regions of the loss landscape. This inherent stability makes them more resilient to the parameter perturbations required during the unlearning process.
Key Results
The empirical results demonstrate a clear advantage for post-grokking models across multiple metrics:
- Superior Performance: Grokked models achieved 6-8% better unlearning effectiveness while maintaining 10-20% higher accuracy on retained data compared to early-stopped versions.
- Efficiency Gains: Post-grokking models reached target forget levels in 5-8 steps, whereas pre-grokking models required 15-20 steps—a 60-70% reduction in computational overhead.
- Privacy and Robustness: Grokked models showed significantly better resistance to Membership Inference Attacks (MIA) and maintained their superior adversarial robustness even after the unlearning process.
Conclusion and Future Work
The "Grokking Tax"—the 5-6x longer training time required—remains a barrier to immediate practical use. However, the paper suggests that when you stop training is an orthogonal lever to how you unlearn. Future research may focus on inducing these beneficial properties (modularity and gradient orthogonality) through better initialization or optimizers like Sharpness-Aware Minimization (SAM), potentially bypassing the need for extended training.
Summary updated