Catastrophic Failure of LLM Unlearning via Quantization

Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, Suhang Wang

Abstract

Large language models (LLMs) have shown remarkable proficiency in generating text, benefiting from extensive training on vast textual corpora. However, LLMs may also acquire unwanted behaviors from the diverse and sensitive nature of their training data, which can include copyrighted and private content. Machine unlearning has been introduced as a viable solution to remove the influence of such problematic content without the need for costly and time-consuming retraining. This process aims to erase specific knowledge from LLMs while preserving as much model utility as possible. Despite the effectiveness of current unlearning methods, little attention has been given to whether existing unlearning methods for LLMs truly achieve forgetting or merely hide the knowledge, which current unlearning benchmarks fail to detect. This paper reveals that applying quantization to models that have undergone unlearning can restore the "forgotten" information. To thoroughly evaluate this phenomenon, we conduct comprehensive experiments using various quantization techniques across multiple precision levels. We find that for unlearning methods with utility constraints, the unlearned model retains an average of 21% of the intended forgotten knowledge in full precision, which significantly increases to 83% after 4-bit quantization.

Introduction

As Large Language Models (LLMs) are trained on massive web-scale datasets, they inevitably ingest sensitive information, including copyrighted works and private data. To address legal requirements like the "Right to be Forgotten" (GDPR) or copyright disputes, developers often turn to machine unlearning. This process modifies a pre-trained model's weights to suppress specific outputs while maintaining general performance, avoiding the prohibitive cost of full retraining.

However, recent research indicates that this "forgetting" is often superficial. Existing unlearning methods typically employ small learning rates and heavy regularization to preserve model utility. This results in minimal weight adjustments that may "hide" knowledge in high-precision (FP32) formats but fail to erase it fundamentally.

Illustration of unlearning failure via quantization. In full precision (FP32), the model refuses to reproduce copyrighted text. After 4-bit quantization, the "forgotten" text is restored.

The Mechanism of Failure: The 4-Bit Cliff

The core vulnerability lies in the interaction between unlearning and quantization—the process of compressing model weights (e.g., from 32-bit to 4-bit) for efficient deployment. Quantization maps continuous weight values into discrete "buckets." Because standard unlearning only shifts weights slightly, these weights often remain within the same quantization interval as the original, "knowledgeable" model. When the model is quantized, the rounding process effectively "snaps" the weights back to their original state, restoring the forgotten information.

This phenomenon is most pronounced at 4-bit precision, which has become the industry standard for local LLM deployment. While 8-bit quantization is relatively stable, 4-bit quantization frequently triggers a "catastrophic" recovery of knowledge.

Empirical Evaluation

The researchers evaluated six representative unlearning methods, including Gradient Ascent (GA) and Negative Preference Optimization (NPO), across benchmarks like MUSE (News and Books) and RWKU (Famous Individuals). The results consistently showed that models appearing to have "forgotten" data in full precision experienced a massive surge in knowledge retention after quantization.

Table showing unlearning performance across NEWS and BOOKS datasets. Note how metrics like M1 (verbatim memorization) skyrocket after 4-bit quantization.

As shown in the experimental data, for methods with utility constraints, knowledge retention jumped from 21% in FP32 to 83% after 4-bit quantization. Even advanced quantization techniques like AWQ and GPTQ failed to prevent this recovery, suggesting the issue is inherent to the "minimal update" philosophy of current unlearning.

Mitigation: The SURE Method

To address this fragility, the authors propose SURE (Saliency-Based Unlearning with a Large Learning Rate). This approach follows three steps:

  1. Identify Salient Modules: Use gradients to pinpoint the specific parameters most responsible for the target knowledge.
  2. Aggressive Updates: Apply a significantly larger learning rate only to these salient modules.
  3. Quantization Robustness: These large updates push weights across quantization boundaries, ensuring the unlearning persists even after compression.

Conclusion

This study serves as a critical wake-up call for the AI safety and legal compliance communities. Machine unlearning, as currently practiced, is often a "thin veil" that is easily pierced by standard deployment techniques like quantization. Developers must validate unlearning effectiveness in the model's final, quantized form to ensure true compliance with privacy and copyright standards.

Our code is available at: https://github.com/zzwjames/FailureLLMUnlearning.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable proficiency in generating text, benefiting from extensive training on vast textual corpora. However, LLMs may also acquire unwanted behaviors from the diverse and sensitive nature of their training data, which can include copyrighted and private content. Machine unlearning has been introduced as a viable solution to remove the influence of such problematic content without the need for costly and time-consuming retraining. This process aims to erase specific knowledge from LLMs while preserving as much model utility as possible. Despite the effectiveness of current unlearning methods, little attention has been given to whether existing unlearning methods for LLMs truly achieve forgetting or merely hide the knowledge, which current unlearning benchmarks fail to detect. This paper reveals that applying quantization to models that have undergone unlearning can restore the "forgotten" information. To thoroughly evaluate this phenomenon, we conduct comprehensive experiments using various quantization techniques across multiple precision levels. We find that for unlearning methods with utility constraints, the unlearned model retains an average of 21\% of the intended forgotten knowledge in full precision, which significantly increases to 83\% after 4-bit quantization. ... Our code is available at: \href{https://github.com/zzwjames/FailureLLMUnlearning}{https://github.com/zzwjames/FailureLLMUnlearning}.
BibTeX
@inproceedings{zhang2025catastrophic,
  author = {Zhiwei Zhang and Fali Wang and Xiaomin Li and Zongyu Wu and Xianfeng Tang and Hui Liu and Qi He and Wenpeng Yin and Suhang Wang},
  title = {{Catastrophic Failure of LLM Unlearning via Quantization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=lHSeDYamnz},
}
Figure