Abstract
Large Language Models (LLMs) have highlighted the necessity of effective unlearning mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims at removing undesired data influences and associated model capabilities without compromising utility beyond the scope of unlearning. While interest in studying LLM unlearning is growing, the impact of the optimizer choice for LLM unlearning remains unexplored. In this work, we shed light on the significance of optimizer selection in LLM unlearning for the first time, establishing a clear connection between second-order optimization and influence unlearning (a classical approach using influence functions to update the model for data influence removal). This insight propels us to develop a second-order optimization-based LLM unlearning framework, termed Second-Order UnLearning (SOUL), which extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process. Our extensive experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics, indicating that second-order optimization offers an effective and broadly applicable solution for LLM unlearning.
Introduction
As Large Language Models (LLMs) become ubiquitous, the need to "unlearn" specific data—due to privacy regulations like GDPR’s "right to be forgotten," copyright infringement, or toxic content—has become critical. Retraining a model from scratch is computationally prohibitive for most organizations. Consequently, "approximate unlearning" techniques have emerged, typically involving fine-tuning the model using first-order (FO) optimizers like AdamW to either maximize loss on a "forget set" or minimize loss on a "retain set."
However, these methods often suffer from two primary issues: over-forgetting, where the model loses general utility (utility collapse), and under-forgetting, where the model retains "ghosts" of the data that remain extractable via prompting or jailbreaks. While research has focused on loss functions, the impact of the optimizer choice has remained largely unexplored until now.
Methodology: The SOUL Framework
The core contribution of this work is SOUL (Second-Order UnLearning), a framework that shifts unlearning from first-order to second-order (SO) optimization.
Connecting Influence Functions to Optimization
The authors mathematically link "influence unlearning"—a classical approach using Hessian matrices to remove data influence—to iterative optimization. They demonstrate that influence unlearning is essentially a single, static second-order step. SOUL extends this concept into a dynamic, iterative process, addressing the accuracy issues inherent in one-shot influence approaches.
Efficient Second-Order Updates
To avoid the $O(d^3)$ computational cost of inverting a full Hessian matrix for $d$ parameters, SOUL leverages efficient Hessian approximations. Specifically, it builds upon the Sophia optimizer, which uses a diagonal Hessian estimate to capture the loss landscape's curvature. SOUL adapts this for unlearning using two modes:
- Ascent Mode: Aggressively increases loss on the forget set.
- Descent Mode: Precisely maintains performance on the retain set.
SOUL is loss-agnostic and can be integrated into existing frameworks such as Gradient Difference (GradDiff), Preference Optimization (PO), and Negative Preference Optimization (NPO).
Experimental Evaluation
The researchers evaluated SOUL across three distinct unlearning tasks:
- TOFU (Task of Fictitious Unlearning): Forgetting 10% of 200 fictitious author profiles while retaining the rest and general world facts.
- Copyrighted Information Removal: Targeted removal of the Harry Potter series from pre-trained models (OPT-1.3B and LLaMA2-7B).
- Model Detoxification: Reducing toxic generation using PKU-SafeRLHF and Real Toxicity Prompts datasets.
Utility was monitored using standard NLP benchmarks including Wikitext, BoolQ, RTE, and HellaSwag to ensure the models remained performant.
Results and Discussion
Experimental results demonstrate that second-order optimization provides significantly more targeted updates than first-order methods. By accounting for curvature, SOUL prevents "catastrophic interference" where forgetting specific facts inadvertently degrades general knowledge.
As shown in the figure above, SO-GradDiff (SOUL) achieved a "Forget Quality" score of 1.00 on the TOFU benchmark, a dramatic improvement over the 0.02 achieved by FO-GradDiff. Qualitatively, SOUL-based models successfully refuse questions about unlearned authors (e.g., "That's outside my area of expertise") while maintaining accurate knowledge of world facts.
The training dynamics further highlight SOUL's effectiveness. Unlike Gradient Ascent (GA) or standard GradDiff, SOUL achieves a rapid and stable reduction in forget accuracy while maintaining high retain accuracy over multiple epochs. This demonstrates the framework's ability to navigate the unlearning objective without compromising the model's underlying capabilities.
Performance and Efficiency
Remarkably, SOUL achieves these gains without significant computational overhead. On the TOFU task, SO-GradDiff required 30 minutes of training—identical to FO-GradDiff—proving that second-order unlearning is practically viable for large-scale models.
Conclusion
SOUL establishes the critical importance of optimizer selection in LLM unlearning. By bridging influence functions with iterative second-order optimization, SOUL offers a robust, efficient, and broadly applicable solution for removing undesired data influences while preserving model utility. Future work will explore the scalability of second-order methods to frontier models exceeding 70B parameters and their robustness against sophisticated adversarial attacks.
Summary updated