Multi-Objective Large Language Model Unlearning

Zibin Pan, Shuwen Zhang, Yuesheng Zheng, Chi Li, Yuheng Cheng, Junhua Zhao

TL;DR

MOLLM (Multi-Objective Large Language Model Unlearning) introduces a stable and surgical way to remove undesirable behaviors from LLMs without the risks of "gradient explosion" or "catastrophic forgetting." By treating unlearning as a multi-objective optimization problem and using a novel dual-space algorithm, it finds a precise update direction that deletes specific data while keeping the model’s general intelligence and fluency intact.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale, they inevitably ingest "toxic" data—harmful responses, copyrighted material, or private user information. The gold standard for fixing this is retraining from scratch, but that is financially and environmentally ruinous.

The community has turned to Machine Unlearning, specifically Gradient Ascent (GA). GA works by essentially telling the model: "Whatever you were going to say here, do the opposite." However, GA faces two massive hurdles:

  1. Gradient Explosion: Because standard Cross-Entropy loss has no upper bound, trying to maximize it (as GA does) can lead to infinite gradients that crash the training process.
  2. Catastrophic Forgetting: In the rush to forget "how to make a bomb," the model might accidentally forget "how to write a poem" or "how to code in Python." Existing fixes, like adding a penalty for deviating from the original model, often create "gradient conflicts" where the goals of forgetting and retaining knowledge pull the model in opposite directions.

Methods

The authors propose MOLLM, which tackles these issues through two main innovations:

1. Unlearning Cross-Entropy (UCE) Loss

Instead of using Gradient Ascent to maximize the standard loss (which is unstable), they design a new Unlearning Cross-Entropy loss. This loss is mathematically bounded, meaning it can be minimized using standard, stable Gradient Descent. It drives the probability of the "forget" tokens toward zero without the risk of the gradients blowing up.

2. Dual Space Multiple Gradient Descent (DS-MGDA)

Rather than just smashing three different losses (Forgetting, KL Divergence, and Retain Performance) into a single weighted sum, MOLLM treats them as independent objectives.

  • The Conflict Problem: In traditional methods, the update for "forgetting" might actively oppose the update for "staying fluent."
  • The Solution: MOLLM uses DS-MGDA to project the gradients of these three objectives into a "dual space." It then calculates a common descent direction—a single vector that simultaneously reduces all three losses. This ensures that every step the model takes helps it forget the bad data while maintaining its original utility.

Datasets

The researchers evaluated MOLLM using the PKU-SafeRLHF dataset. This is a specialized safety alignment dataset containing:

  • Harmful questions and responses (the "forget" set).
  • Non-harmful, helpful responses (the "retain" set).

They tested their approach on the state-of-the-art Llama3-8B model.

Key Insights and Ideas

  • Stop Fighting Gradients: The "aha moment" is realizing that unlearning fails because different goals are fighting for control over the model's weights. By finding the "common ground" update direction, you eliminate the tug-of-war.
  • Boundedness is Safety: Moving from Gradient Ascent (unbounded) to a custom Unlearning Loss (bounded) removes the need for manual "gradient clipping" and tedious hyperparameter tuning.
  • Pareto Stationarity: The method aims for a state where you cannot make the model forget more without hurting its general performance, representing an optimal trade-off.

Key Results

MOLLM was compared against several baselines, including standard GA and Orthogonal Gradient Descent (OGD).

  • Lower Toxicity: MOLLM achieved a Harmful Rate of 3.45%, outperforming GA + Weighted-sum (5.46%) and GA + OGD (3.68%).
  • Superior Fluency: While other methods often made the model's text less coherent (increasing the fluency score), MOLLM actually improved fluency relative to the original model (1.689 vs 1.754).
  • Zero Conflict: The authors measured the "probability of conflict" between objectives. In MOLLM, this probability was 0%, meaning the algorithm successfully found a direction that satisfied all goals in every update step.

Limitations and Future Work

  • Computational Overhead: While DS-MGDA is efficient because it uses fundamental matrix computations, calculating three separate gradients per step is still more intensive than a single-objective update.
  • Scaling to Massive Deletions: The study focuses on safety alignment (forgetting harmful behaviors). It remains to be seen how this multi-objective approach handles "bulk unlearning," such as removing an entire book series or a massive codebase from a model's memory.
  • Matrix Rank: The DS-MGDA algorithm assumes the gradient matrix is full-rank; the authors note that if the objectives become too similar, the projection might require additional handling.

Summary updated

Abstract

Machine unlearning in the domain of large language models (LLMs) has attracted great attention recently, which aims to effectively eliminate undesirable behaviors from LLMs without full retraining from scratch. In this paper, we explore the Gradient Ascent (GA) approach in LLM unlearning, which is a proactive way to decrease the prediction probability of the model on the target data in order to remove their influence. We analyze two challenges that render the process impractical: gradient explosion and catastrophic forgetting. To address these issues, we propose Multi-Objective Large Language Model Unlearning (MOLLM) algorithm. We first formulate LLM unlearning as a multi-objective optimization problem, in which the cross-entropy loss is modified to the unlearning version to overcome the gradient explosion issue. A common descent update direction is then calculated, which enables the model to forget the target data while preserving the utility of the LLM. Our empirical results verify that MoLLM outperforms the SOTA GA-based LLM unlearning methods in terms of unlearning effect and model utility preservation. The source code is available at https://github.com/zibinpan/MOLLM.
BibTeX
@inproceedings{pan2025multi,
  author = {Zibin Pan and Shuwen Zhang and Yuesheng Zheng and Chi Li and Yuheng Cheng and Junhua Zhao},
  title = {{Multi-Objective Large Language Model Unlearning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889776},
}

Loading related content...

Figure