Abstract
Machine unlearning in the domain of large language models (LLMs) has attracted great attention recently, which aims to effectively eliminate undesirable behaviors from LLMs without full retraining from scratch. In this paper, we explore the Gradient Ascent (GA) approach in LLM unlearning, which is a proactive way to decrease the prediction probability of the model on the target data in order to remove their influence. We analyze two challenges that render the process impractical: gradient explosion and catastrophic forgetting. To address these issues, we propose Multi-Objective Large Language Model Unlearning (MOLLM) algorithm. We first formulate LLM unlearning as a multi-objective optimization problem, in which the cross-entropy loss is modified to the unlearning version to overcome the gradient explosion issue. A common descent update direction is then calculated, which enables the model to forget the target data while preserving the utility of the LLM. Our empirical results verify that MoLLM outperforms the SOTA GA-based LLM unlearning methods in terms of unlearning effect and model utility preservation.
Introduction
As Large Language Models (LLMs) scale, they inevitably ingest "toxic" data—harmful responses, copyrighted material, or private user information. While retraining from scratch is the gold standard for data removal, it is financially and environmentally prohibitive. This has led to the rise of Machine Unlearning, specifically Gradient Ascent (GA). GA attempts to remove influence by maximizing the loss on target data. However, GA faces two critical hurdles:
- Gradient Explosion: Standard Cross-Entropy loss lacks an upper bound; maximizing it can lead to infinite gradients that destabilize training.
- Catastrophic Forgetting: Efforts to forget specific harmful data often inadvertently degrade the model's general capabilities (e.g., coding or reasoning). Existing solutions often create "gradient conflicts" where the goals of forgetting and retaining knowledge pull the model in opposite directions.
Methodology
MOLLM addresses these challenges through two primary innovations: Unlearning Cross-Entropy (UCE) Loss and Dual Space Multiple Gradient Descent (DS-MGDA).
Unlearning Cross-Entropy (UCE) Loss
Instead of maximizing standard loss, MOLLM utilizes a mathematically bounded Unlearning Cross-Entropy loss. This allows the unlearning process to be framed as a stable minimization problem, driving the probability of "forget" tokens toward zero without the risk of gradient explosion.
Dual Space Multiple Gradient Descent (DS-MGDA)
MOLLM treats forgetting, KL Divergence (to maintain the original distribution), and retain performance as independent objectives. Traditional weighted-sum methods often suffer from objective conflict. MOLLM employs DS-MGDA to project these gradients into a "dual space" to find a common descent direction.
As shown in the figure above, the algorithm identifies a single vector ($d_{MOLLM}$) that simultaneously reduces all objective losses, ensuring that every update step contributes to forgetting without sacrificing model utility.
Experimental Setup
The researchers evaluated MOLLM using the PKU-SafeRLHF dataset, which contains harmful questions (the "forget" set) and helpful responses (the "retain" set). The experiments were conducted on the Llama3-8B model.
Results
MOLLM demonstrates superior performance across safety and utility metrics compared to baselines like standard GA and Orthogonal Gradient Descent (OGD).
Key findings include:
- Lower Toxicity: MOLLM achieved a Harmful Rate (HR) of 3.45%, outperforming GA + Weighted-sum (5.46%).
- Utility Preservation: MOLLM maintained superior fluency (1.689) compared to the original model (1.754), whereas other methods often caused coherence degradation.
- Zero Conflict: As indicated by the $PC$ metrics in the results table, MOLLM successfully eliminated objective conflict ($0\%$ probability of conflict), confirming the effectiveness of the common descent direction approach.
Conclusion and Future Work
MOLLM provides a stable, surgical method for LLM unlearning by resolving the tug-of-war between forgetting and retention. By moving to a bounded loss and a multi-objective optimization framework, it achieves Pareto stationarity—an optimal trade-off where forgetting is maximized without harming general performance.
Future work will explore the computational overhead of calculating multiple gradients and the scalability of this approach to "bulk unlearning," such as removing massive proprietary codebases or entire literary corpora.
Summary updated