Machine unlearning is the process of removing specific information—such as private data, copyrighted text, or harmful biases—from a pre-trained model without retraining it from scratch. While the concept seems straightforward (simply "reverse" the learning process), the reality is mathematically treacherous.
Recent research reveals that the most intuitive approach to unlearning often leads to "catastrophic collapse," where a model loses its general intelligence while trying to forget a single fact. To solve this, researchers are moving away from naive error maximization toward stabilized, bounded optimization strategies.
The Divergence Problem of Gradient Ascent
The most common heuristic for unlearning is Gradient Ascent (GA). If training involves minimizing a loss function (like cross-entropy) to make a model's predictions more accurate, unlearning should involve maximizing that same loss to make the model "forget."
However, as Simon Lucey and Arpit Garg demonstrate in Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs, GA is fundamentally unstable. In standard cross-entropy loss, there is no global maximum. When you tell an optimizer to maximize loss, it attempts to drive the model's prediction probabilities to zero. Mathematically, this forces the model's weights toward infinity. This "weight explosion" destroys the delicate parameter structure that allows a Large Language Model (LLM) to reason or speak coherently.
Ioannis Mavrothalassitis and colleagues further argue in Ascent Fails to Forget that because the data to be forgotten is often statistically linked to the data we want to keep, pushing the model "away" from the forget set inevitably drags it away from the optimal state for the retain set.
Bounding the Loss to Prevent Collapse
To prevent weights from drifting into infinity, researchers have introduced bounded loss functions. The intuition is to give the optimizer a "stopping point"—a level of confusion that is sufficient for forgetting but not so extreme that it breaks the model.
- Lower-Bounded Loss: Jie Chen and Yupeng Zhang propose a surgical unlearning strategy in Unveiling the Flaws that uses a lower-bounded forgetting loss. This prevents the loss from becoming unmanageable as prediction probabilities approach zero.
- Bounded Activation Functions: Instead of changing the loss, Simon Lucey suggests applying bounded functions like sine or tanh to LoRA adapters. This physically constrains the parameter updates, ensuring that even if the gradient ascent is aggressive, the weights cannot explode.
- Unlearning Cross-Entropy: In the context of Federated Learning, Zibin Pan and Chi Li introduced FedOSD in Federated Unlearning with Gradient Descent and Conflict Mitigation. Their "Unlearning Cross-Entropy" loss is specifically designed to be upper-bounded, preventing the gradient explosion that typically occurs when a client's data is removed from a global model.
Anchoring to a Safe State
If unlearning is a journey away from "bad" data, the model needs an anchor to ensure it doesn't wander into nonsensical territory. Several recent papers propose using "safe" data or "teacher" models as reference points.
Zirui Pang and Jiaheng Wei suggest Smoothed Gradient Ascent (SGA) in Label Smoothing Improves Gradient Ascent in LLM Unlearning. By mixing the "forget" data with "safe" normal data and applying label smoothing, they stabilize the optimization path. The safe data acts as a constraint, reminding the model how to behave while it erases specific knowledge.
Similarly, Yegor Klochkov introduces a Mean Teacher approach in A mean teacher algorithm for unlearning of language models. By maintaining a moving average of the model's weights (the teacher), the unlearning process (the student) is penalized if it deviates too far from the teacher's general capabilities. This mimics the stability of Natural Gradient Descent, providing a "soft" boundary for the forgetting process.
Matching the Retrained Oracle
The "gold standard" for unlearning is a model retrained from scratch without the target data. Since retraining is expensive, researchers are finding ways to simulate its results.
In Machine Unlearning via Simulated Oracle Matching, Kristian Georgiev and Andrew Ilyas propose Datamodel Matching (DMM). Instead of maximizing error on the forget set, DMM uses data attribution to predict what a retrained model would have outputted. The original model is then fine-tuned to match these predicted "oracle" outputs. This provides a fixed convergence point, solving the instability of open-ended gradient ascent.
Surgical Subspace Suppression
Finally, some researchers argue that we shouldn't use gradients at all. If unlearning is about removing specific concepts, we can identify the mathematical "space" those concepts occupy and suppress them.
Kaushik Roy and Gobinda Saha developed a training-free approach in Deep Unlearning. By using Singular Value Decomposition (SVD), they identify the activation spaces associated with specific classes and suppress them in a single weight update. This "gradient-free" method bypasses the stability issues of iterative optimization entirely, offering a fast and precise way to "confuse" the model about specific targets without affecting the rest of its knowledge.