Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo

Abstract

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for LMs has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger-sized LMs. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with previous methods known to mitigate privacy risks for LMs, we show that our approach can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust.

Introduction: The "Sponge" Problem in LLMs

Large Language Models (LLMs) are notorious for acting like sponges, soaking up vast amounts of training data that often include sensitive personal information (PII) such as home addresses, phone numbers, and private notes. This memorization poses significant privacy risks and legal challenges, particularly with the emergence of "Right to Be Forgotten" (RTBF) legislation.

Historically, the primary solutions have been computationally expensive and often degrade model utility:

  1. Data Preprocessing: Scrubbing the training set and retraining the model from scratch—a process costing millions of dollars.
  2. Differential Privacy (DP): Training with noise to prevent memorization, which often leads to a significant "utility gap."
Comparison of privacy mitigation approaches: retraining after sanitization, retraining with DP, and the proposed knowledge unlearning approach.

As illustrated in Figure 1, the proposed knowledge unlearning approach offers a "delete button" that is millions of times more computationally efficient than traditional retraining methods, requiring only a fraction of the compute (~0.001 A100 GPU days vs. 900-1800 days).

Methodology: Surgical Forgetting via Gradient Ascent

The core technique is conceptually simple: Gradient Ascent. While standard training uses gradient descent to minimize loss and improve prediction, unlearning uses gradient ascent to maximize loss for specific target sequences. This effectively teaches the model to be as poor as possible at predicting that specific private information.

Sequential vs. Batch Unlearning

A key methodological insight is that forgetting is a delicate process. The researchers found that Sequential Unlearning—processing data in smaller, sequential chunks—is far more stable than Batch Unlearning (trying to forget a large amount of data at once), which tends to degrade the model's general performance.

New Privacy Metrics

To quantify success, the paper introduces two key metrics:

  • Extraction Likelihood (EL): Quantifies privacy risk by measuring how much of a "suffix" (private info) a model can generate given a "prefix" (context).
  • Memorization Accuracy (MA): Tracks how many tokens the model gets exactly right for sensitive sequences.

Key Results and Performance

The researchers evaluated their method using the GPT-Neo family (125M to 2.7B parameters) against a variety of classification and dialogue benchmarks.

Main results comparing the original model (NEO) against Differential Privacy Decoding (DPD+) and the proposed Unlearning (UL/UL+) methods across different model sizes.

As detailed in Figure 6, the unlearning approach (+UL) successfully reduced privacy risks (EL and MA) to levels comparable to models that had never seen the data (OPT baseline). Crucially, it maintained high performance on general tasks. Larger models (2.7B) proved to be "stronger unlearners," retaining their general reasoning capabilities much better than smaller models after the unlearning process.

Insights and Future Directions

  • Unlearning as Regularization: Surprisingly, unlearning certain "noisy" or highly memorized sequences actually improved performance on some benchmarks, suggesting a post-hoc regularization effect.
  • Domain Sensitivity: Structured data, such as code or licenses, is significantly easier for the model to forget than unstructured natural language text.
  • Efficiency: For a 2.7B parameter model, unlearning was approximately 3.5 million times more computationally efficient than retraining from scratch.

While effective against naive extraction, future work must address more sophisticated adversarial attacks and ensure that unlearning one individual's data does not inadvertently expose another's—a phenomenon known as the "Privacy Onion" effect.

Summary updated

Abstract

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for LMs has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger-sized LMs. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with previous methods known to mitigate privacy risks for LMs, we show that our approach can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust.
BibTeX
@inproceedings{jang2023knowledge,
  author = {Joel Jang and Dongkeun Yoon and Sohee Yang and Sungmin Cha and Moontae Lee and Lajanugen Logeswaran and Minjoon Seo},
  title = {{Knowledge Unlearning for Mitigating Privacy Risks in Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=TkVue8LoYe},
}
Figure