Stable Machine Unlearning through Bounded Objectives

This direction is merged into another direction: catastrophic forgetting in unlearning. Its last version stays readable and citable here.

Version 1 of 119 papers28 conceptsfrozen 2026-10-04

Machine unlearning—the process of removing specific data or concepts from a trained model—is notoriously unstable. The most intuitive approach is "Gradient Ascent": if training involves minimizing a loss function to learn, unlearning should involve maximizing that same loss to forget. However, this often leads to "model collapse," where the model begins generating gibberish or loses its general reasoning capabilities.

Recent research has shifted away from divergent maximization toward stable, bounded minimization. By reframing unlearning as a convergent optimization problem, researchers are finding ways to erase specific memories without destroying the model's underlying intelligence.

The Instability of Gradient Ascent

To understand why unlearning is difficult, we must look at the geometry of the loss landscape. In standard cross-entropy training, the loss is $-\log(p)$, where $p$ is the probability of the correct token. As the model learns, $p$ approaches 1 and the loss approaches 0.

If we use gradient ascent to unlearn, we try to maximize this loss. As Zibin Pan and Sungmin Cha point out, this is numerically dangerous. As the probability $p$ of the "forgotten" data approaches zero, the gradient of the log-loss explodes toward infinity. This "gradient explosion" pushes model weights into chaotic regions of the parameter space, leading to the catastrophic loss of unrelated knowledge.

Replacing Ascent with Bounded Minimization

The core insight across recent work is that unlearning should be a minimization problem toward a "safe" target, rather than a maximization problem away from a "bad" one.

Jiabao Ji and colleagues demonstrate this in Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference. Instead of forcing the main model to forget, they train a small "assistant" model to learn the target data. They then subtract the assistant's logits from the main model. Because the assistant is performing standard (stable) minimization, the entire process avoids the numerical instability of gradient ascent.

Similarly, Hongzhou Lin introduces UNDIAL in Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models. This method uses self-distillation to pull the model toward a modified version of its own output. By minimizing the distance to a "sanitized" distribution rather than maximizing the distance from the original one, the model stays within a stable optimization trajectory.

Designing Bounded Loss Functions

Another strategy involves modifying the loss function itself to ensure gradients remain manageable even as the model "forgets."

Preserving the Retain Distribution

A final pillar of stable unlearning is ensuring that the "forget" operation doesn't bleed into the "retain" set. Shizhou Xu argues in Machine Unlearning via Information Theoretic Regularization that unlearning should be framed as a regularization problem. By explicitly targeting a specific "retain distribution," the model is anchored to its useful knowledge while the sensitive information is pruned.

This is echoed by Cheng Jingpu, who addresses "entanglement"—the phenomenon where deleting one fact accidentally deletes a related, safe fact. Their work on Machine Unlearning under Retain–Forget Entanglement suggests that preserving the distribution of losses, rather than just the average, is key to maintaining model integrity.

Summary of the Shift

The transition from Gradient Ascent to Bounded Minimization represents a maturation of the field. By recognizing that the primary enemy of unlearning is numerical instability, researchers like Jiabao Ji, Sungmin Cha, and Zibin Pan have moved toward frameworks that treat forgetting as a controlled, convergent process. This ensures that when a model "forgets," it does so with a whisper rather than a crash.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stable_machine_unlearning_through_bounde_03b94966,
  title = {Stable Machine Unlearning through Bounded Objectives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stable-machine-unlearning-through-bounded-objectives/rdr_03b94966306c493ab0733f5fb482ccdd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_stable_machine_unlearning_through_bounde_03b94966_v1,
  title = {Stable Machine Unlearning through Bounded Objectives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stable-machine-unlearning-through-bounded-objectives/rdr_03b94966306c493ab0733f5fb482ccdd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}