When we train a machine learning model, our goal is straightforward: minimize a loss function so the model learns to predict the correct outputs. But what happens when we want a model to forget?
Intuitively, one might think we can simply flip the sign of the optimization objective. If minimizing cross-entropy loss makes a model memorize a dataset, then maximizing that same loss via gradient ascent should force it to unlearn sensitive, copyrighted, or toxic information.
However, this intuitive leap hides a severe mathematical trap. Across deep learning—and particularly within Large Language Models (LLMs)—naive gradient ascent frequently triggers an optimization failure known as catastrophic collapse or gradient explosion. Instead of cleanly erasing specific facts, the model's weights diverge, its internal representations shatter, and its output degrades into repetitive loops or complete gibberish.
To understand why this happens and how to fix it, we must look closely at the geometry of the loss landscapes we create when we try to force a model to forget.
The Core Dilemma: The Unbounded Loss Landscape
The fundamental issue with using standard cross-entropy loss for gradient ascent is that the objective function is mathematically unbounded.
When minimizing cross-entropy during standard training, the loss has a natural global minimum at zero, corresponding to a probability of $1.0$ for the target token. The optimization process naturally slows down and stabilizes as it approaches this floor.
Conversely, when we perform gradient ascent to maximize cross-entropy, we are pushing the model to assign a probability of $0.0$ to the target text. As Zibin Pan et al. point out in their work on Multi-Objective Large Language Model Unlearning, the cross-entropy loss function is not upper-bounded. As the target token's probability approaches zero, the loss approaches infinity.
Standard Training (Minimization) Naive Unlearning (Maximization)
| ^
| | (Loss -> Infinity)
v |
[ Loss Floor: 0.0 ] [ No Upper Bound! ]
This lack of an upper bound creates an unstable gradient field. As the optimization process aggressively drives the target logits toward negative infinity, the resulting gradients explode. As documented by Simon Lucey et al. in Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs, this lack of a global minimum forces weight and gradient magnitudes to diverge toward infinity. The model's weights shift so drastically that its underlying language distribution is destroyed, causing extreme spikes in perplexity and a total breakdown of linguistic fluency—a phenomenon often referred to as the "lobotomy" problem (Yu Wang et al.).
Furthermore, this trajectory lacks a natural stopping point. As explored by Dongkeun Yoon et al., gradient ascent lacks a robust, task-agnostic convergence metric. Without an explicit anchor, running the optimization for even a few iterations too many causes the model's general reasoning capabilities to rapidly evaporate (Kristian Georgiev et al.; Hongzhou Lin et al.).
Strategy 1: Bounding the Loss Landscape
If the root cause of the instability is an unbounded loss function, the most direct solution is to engineer a ceiling into the objective. By capping the maximum penalty an unlearned sample can incur, we can prevent gradients from exploding once a fact is "sufficiently forgotten."
Negative Preference Optimization (NPO)
Borrowing insights from human preference alignment, Ruiqi Zhang et al. introduced Negative Preference Optimization (NPO). Instead of letting the loss grow infinitely as the target token probability drops, NPO uses a bounded, sigmoid-based objective similar to Direct Preference Optimization (DPO).
The intuition is elegant: once the model's probability of generating the sensitive data drops significantly below its original baseline, the loss saturates, and the gradient drops to zero. This prevents the optimization from driving the weights to infinity, allowing the model to forget up to 90% of its training data while preserving its general intelligence.
Inverted Hinge Loss (IHL)
Another way to construct a mathematical ceiling is through a hinge mechanism. In Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs, Sungmin Cha et al. propose an Inverted Hinge Loss (IHL). IHL explicitly sets a threshold for forgetting. If the model's likelihood of generating the target sequence is higher than the threshold, it receives a penalty; however, the moment the likelihood falls below this safety margin, the loss becomes exactly zero. This explicit constraint completely halts the weight drift, ensuring the model remains anchored within its functional parameter space.
Strategy 2: Reframing Unlearning as Minimization
A second major paradigm shifts the optimization direction entirely. If maximizing error is inherently unstable, can we reformulate unlearning as a minimization problem?
Instead of telling the model what not to do (which is open-ended and unbounded), these approaches give the model a specific, stable target to move toward using standard gradient descent.
Naive Ascent: [Current Model] --------(Maximize Loss)--------> [Unbounded Void]
Reframed Descent: [Current Model] --------(Minimize Distance)------> [Stable Target/Oracle]
Unlearning from Logit Difference (ULD)
The authors of Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (Jiabao Ji et al.) implement this by training a small, temporary "assistant" model. The assistant is intentionally trained to learn the target data while forgetting general knowledge.
By subtracting the assistant's logits from the main model's logits, the framework creates a stable target distribution. The main model is then trained to minimize its distance to this adjusted logit distribution. Because the objective is a standard minimization task, the optimization process is highly stable, eliminating gibberish outputs and preventing the destruction of general knowledge.
Datamodel Matching (DMM)
A similar intuition drives the Datamodel Matching (DMM) framework proposed by Kristian Georgiev et al.. Instead of guessing how to push weights away from the forgotten data, the authors use data attribution to simulate the exact output distribution of an "oracle" model—a hypothetical model that was completely retrained from scratch without the forbidden data. The original model is then fine-tuned via standard gradient descent to match these simulated targets, providing a fixed, stable convergence point.
Strategy 3: Structural and Parameter Constraints
Rather than modifying the loss function, a third approach to stabilizing unlearning is to place physical constraints on the parameters themselves, preventing them from exploding even if the loss function demands it.
While Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA are widely used to save memory, standard LoRA adapters are still vulnerable to unbounded weight growth when subjected to inverse optimization objectives. To counteract this, Simon Lucey et al. introduced a structural modification: applying bounded activation functions (such as sine or tanh) directly to the LoRA adapter weights.
$$\Delta W = \text{tanh}(A) \cdot B$$
Because the tanh or sine functions have strict horizontal asymptotes (capping values between -1 and 1), the adapter parameters physically cannot grow toward infinity. Even if gradient ascent attempts to drive the updates aggressively, the mathematical structure of the adapter acts as a governor, absorbing the shock and enforcing stability throughout the forgetting process.
Conclusion
The transition from making a model learn to making it forget is not as simple as changing a plus sign to a minus sign. Naive gradient ascent breaks the implicit guardrails of deep learning by turning a bounded minimization problem into an unbounded maximization race toward infinity.
Whether by engineering explicit ceilings into the loss function (NPO, IHL), reframing the task back into a stable minimization problem (ULD, DMM), or placing physical boundaries on the parameter space itself, stabilizing the optimization landscape is the key to successful machine unlearning. By providing a clear, bounded destination, these techniques allow models to cleanly excise targeted data while keeping their core intelligence intact.
Key Foundational Work
Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (2024) — contributed 2 concept elements to this direction.
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024) — contributed 1 concept elements to this direction.
Multi-Objective Large Language Model Unlearning (2024) — contributed 1 concept elements to this direction.