Machine Unlearning via Targeted Gradient Ascent

39 concepts36 papers1 proposal

When we train a deep learning model, we use gradient descent to guide its parameters down a loss landscape, carving out deep valleys (local minima) where the model predicts the training data with high confidence. But what happens when a model memorizes something it shouldn't have—such as private personal information, copyrighted material, or toxic behavior?

Retraining a massive model from scratch without the problematic data is often computationally impossible. Machine unlearning solves this by doing the exact opposite of training: running optimization in reverse via gradient ascent. By calculating the gradient on a targeted "forget set" and moving the model's weights away from it, we can actively flatten the specific valleys of memorization, inducing targeted ignorance while leaving the rest of the model's knowledge intact.

The Geometry of Forgetting

To understand why gradient ascent works, consider the parameter space of a trained model. The model has learned to associate specific inputs with specific outputs because its weights reside in a low-loss basin for those pairs. As Joel Jang et al. demonstrate in Knowledge Unlearning for Mitigating Privacy Risks in Language Models, applying gradient ascent to maximize the loss function on sensitive target sequences directly reduces the model's probability of generating that information. Instead of trying to overwrite the data, the optimization process pushes the parameters up and out of that specific basin.

This intuition is backed by rigorous theory. In Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks, Odelia Melamed et al. prove that even a single gradient ascent step can successfully shift model weights away from a targeted sample's representation. By leveraging the implicit bias of gradient optimization, this single step forces the network into a state that closely approximates a model retrained entirely from scratch without the deleted data.

Mitigating Collateral Damage

The primary challenge of gradient ascent is precision. If you blindly push a model's weights up the loss hill, you risk destroying adjacent valleys that hold valuable, unrelated knowledge—a phenomenon known as model collapse or catastrophic forgetting. Researchers have developed several mechanisms to confine the unlearning effect to the target data:

Reshaping the Loss Landscape

A persistent concern in machine unlearning is whether the target information is truly erased or merely obscured. As Abhinav Joshi et al. point out in Towards Robust Evaluation of Unlearning in LLMs via Data Transformations, naive unlearning can sometimes leave the underlying knowledge intact, allowing it to be extracted if the prompt format is slightly altered.

To ensure permanent erasure, Haoran Tang et al. introduce a sharpness-aware framework in Sharpness-Aware Machine Unlearning. By intentionally maximizing the sharpness of the loss landscape for the forget set, they force the model to overfit to a forgetting signal. This completely destroys the unique features associated with those targeted samples.

This strategy of structural disruption is also highly effective for safety alignment. In Representation Noising: A Defence Mechanism Against Harmful Finetuning, David Atanasov et al. use gradient ascent to strip away the underlying information structure of harmful concepts across all model layers. This makes it exceptionally difficult for an adversary to repurpose the model for malicious ends, even via subsequent fine-tuning.

Applications Beyond Privacy

While privacy and safety are the primary drivers of this research, targeted gradient ascent has surfaced as a versatile tool across several machine learning subfields:

Key Foundational Work

Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023) — contributed 1 concept elements to this direction.
SIGUA: Forgetting May Make Learning with Noisy Labels More Robust (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_ascent_unlearning_d9823a17,
  title = {gradient ascent unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-ascent-unlearning/rdr_d9823a176c98429d86e49c952ca78fda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_ascent_unlearning_d9823a17_v1,
  title = {gradient ascent unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-ascent-unlearning/rdr_d9823a176c98429d86e49c952ca78fda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake