When we want a machine learning model to "forget" a specific subset of its training data—whether to protect privacy, remove copyrighted material, or eliminate harmful behaviors—the most intuitive approach is to reverse the training process. This traditional method, known as Gradient Ascent (GA), takes the standard cross-entropy loss function and flips its sign to minimize the likelihood of generating the target data.
However, GA suffers from a fundamental mathematical flaw: it is an unconstrained, unbounded optimization problem. As the model attempts to drive the probability of the target tokens to absolute zero, the cross-entropy loss approaches infinity. This forces the model's weights to change drastically and unpredictably. As documented by Chongyu Fan et al. in LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics, this unconstrained ascent often triggers a "lobotomy" effect or catastrophic collapse, where the model's general reasoning capabilities and linguistic coherence are completely destroyed.
Framing Unlearning as a Preference Problem
To solve this instability, researchers have shifted the paradigm from violent erasure to behavioral alignment. Instead of trying to force the probability of a sequence to zero, what if we simply teach the model that the sequence is highly dispreferred?
This is the core intuition behind Negative Preference Optimization (NPO), a framework introduced by Ruiqi Zhang et al. in Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. NPO adapts the mathematical foundations of Direct Preference Optimization (DPO)—a method widely used to align models with human preferences—but tailors it for scenarios where only negative or undesirable samples are available.
By framing unlearning as a preference task, NPO treats the data to be forgotten as a dispreferred response. As noted by Soumyadeep Pal et al. in LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks, this formulation actively pushes the model's output distribution away from the target data without requiring a paired set of preferred "correct" answers.
Why NPO Works: Bounded Gradients and Probability Reallocation
To understand why NPO prevents catastrophic collapse while GA causes it, we must look at how the gradients behave during optimization. NPO replaces the unbounded cross-entropy loss with a sigmoid-based loss function formulated over the relative likelihood of the sequence.
In standard Gradient Ascent, the gradient grows larger and more volatile the harder the model tries to forget, driving weights toward infinity. In contrast, the sigmoid function in NPO acts as a natural governor. As the probability of the forbidden data decreases, the sigmoid function saturates, causing the gradient to smoothly taper off to zero. As demonstrated by Qiang Chen et al. in Graph Unlearning Meets Influence-aware Negative Preference Optimization, this bounded nature introduces an implicit adaptive coefficient that slows down the speed of model divergence, stabilizing the entire optimization landscape.
The second reason NPO succeeds is how it handles the model's internal probability mass. When GA suppresses a token sequence, it leaves a chaotic vacuum in the probability distribution, leading to incoherent outputs. NPO, however, acts as a smooth reallocation mechanism. As explained by Keivan Rezaei et al. in RESTOR: Knowledge Recovery in Machine Unlearning, preference-based objectives encourage the model to re-allocate its probability mass away from the target data and back toward the original, healthy knowledge distribution. Alternatively, as shown by Hwan Chang et al. in Which Retain Set Matters for LLM Unlearning? A Case Study on Entity Unlearning, it can smoothly steer the model toward neutral or uncertain responses (such as stating it does not know the answer) rather than breaking the model's linguistic fluency.
Applications Across Modalities and Safety Frameworks
Because NPO operates on the fundamental geometry of the model's preference landscape, it has proven highly versatile across diverse unlearning challenges:
- Entity-Level Unlearning: Erasing an entire real-world entity requires scrubbing interconnected facts without collateral damage to surrounding knowledge. Minseok Choi et al. leverage this in Opt-Out: Investigating Entity-Level Unlearning for Large Language Models via Optimal Transport to minimize target generation probabilities stably, while Weitao Ma et al. in Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis use negative signals to gracefully transition queries about forgotten entities into neutral responses.
- Active Safety Alignment: Traditional safety training can be bypassed by clever jailbreaks. To counter this, Xuandong Zhao et al. developed the DOOR framework in Improving LLM Safety Alignment with Dual-Objective Optimization, which uses NPO to actively penalize unsafe trajectories during training, forcing the model to unlearn harmful patterns rather than merely masking them.
- Beyond Text: The divergence-driven mechanics of NPO are not restricted to language. Aleksej Dontsov et al. in CLEAR: Character Unlearning in Textual and Visual Modalities adapt preference optimization to penalize sensitive outputs across both text and image modalities, ensuring that the model's latent space remains clean across multiple domains.
Key Foundational Work
Graph Unlearning Meets Influence-aware Negative Preference Optimization (2025) — contributed 2 concept elements to this direction.
LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks (2025) — contributed 1 concept elements to this direction.
CLEAR: Character Unlearning in Textual and Visual Modalities (2025) — contributed 1 concept elements to this direction.