Machine unlearning—the process of removing specific data or concepts from a trained model—is notoriously unstable. The most intuitive approach is "Gradient Ascent": if training involves minimizing a loss function to learn, unlearning should involve maximizing that same loss to forget. However, this often leads to "model collapse," where the model begins generating gibberish or loses its general reasoning capabilities.
Recent research has shifted away from divergent maximization toward stable, bounded minimization. By reframing unlearning as a convergent optimization problem, researchers are finding ways to erase specific memories without destroying the model's underlying intelligence.
The Instability of Gradient Ascent
To understand why unlearning is difficult, we must look at the geometry of the loss landscape. In standard cross-entropy training, the loss is $-\log(p)$, where $p$ is the probability of the correct token. As the model learns, $p$ approaches 1 and the loss approaches 0.
If we use gradient ascent to unlearn, we try to maximize this loss. As Zibin Pan and Sungmin Cha point out, this is numerically dangerous. As the probability $p$ of the "forgotten" data approaches zero, the gradient of the log-loss explodes toward infinity. This "gradient explosion" pushes model weights into chaotic regions of the parameter space, leading to the catastrophic loss of unrelated knowledge.
Replacing Ascent with Bounded Minimization
The core insight across recent work is that unlearning should be a minimization problem toward a "safe" target, rather than a maximization problem away from a "bad" one.
Jiabao Ji and colleagues demonstrate this in Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference. Instead of forcing the main model to forget, they train a small "assistant" model to learn the target data. They then subtract the assistant's logits from the main model. Because the assistant is performing standard (stable) minimization, the entire process avoids the numerical instability of gradient ascent.
Similarly, Hongzhou Lin introduces UNDIAL in Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models. This method uses self-distillation to pull the model toward a modified version of its own output. By minimizing the distance to a "sanitized" distribution rather than maximizing the distance from the original one, the model stays within a stable optimization trajectory.
Designing Bounded Loss Functions
Another strategy involves modifying the loss function itself to ensure gradients remain manageable even as the model "forgets."
- Inverted Hinge Loss (IHL): Sungmin Cha proposes IHL in Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs. Unlike log-loss, which is unbounded, IHL provides a flat plateau once the probability of the forgotten data drops below a certain threshold. This prevents the model from "over-forgetting" and destabilizing its weights.
- Bounded Cross-Entropy: In the context of federated learning, Zibin Pan uses a bounded "Unlearning Cross-Entropy" in Federated Unlearning with Gradient Descent and Conflict Mitigation. This ensures that the updates intended to remove one client's data do not conflict with or destroy the knowledge shared by others.
- f-Divergences: For diffusion models, Nicola Novello shows in A Unified Framework for Diffusion Model Unlearning with f-Divergence that using the Hellinger distance—which has naturally bounded gradients—preserves image quality much better than standard Mean Squared Error during the erasure of concepts.
Preserving the Retain Distribution
A final pillar of stable unlearning is ensuring that the "forget" operation doesn't bleed into the "retain" set. Shizhou Xu argues in Machine Unlearning via Information Theoretic Regularization that unlearning should be framed as a regularization problem. By explicitly targeting a specific "retain distribution," the model is anchored to its useful knowledge while the sensitive information is pruned.
This is echoed by Cheng Jingpu, who addresses "entanglement"—the phenomenon where deleting one fact accidentally deletes a related, safe fact. Their work on Machine Unlearning under Retain–Forget Entanglement suggests that preserving the distribution of losses, rather than just the average, is key to maintaining model integrity.
Summary of the Shift
The transition from Gradient Ascent to Bounded Minimization represents a maturation of the field. By recognizing that the primary enemy of unlearning is numerical instability, researchers like Jiabao Ji, Sungmin Cha, and Zibin Pan have moved toward frameworks that treat forgetting as a controlled, convergent process. This ensures that when a model "forgets," it does so with a whisper rather than a crash.