Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo

Abstract: Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust.

Introduction

Large Language Models (LLMs) are notorious for "memorizing" their training data, which often includes Personally Identifiable Information (PII) such as home addresses, phone numbers, and private clinical notes. As models scale, this memorization becomes more pronounced, creating significant privacy risks and legal challenges regarding the "Right To Be Forgotten" (RTBF).

Traditionally, addressing these risks required either Data Preprocessing (identifying and removing sensitive data before training) or Differential Privacy (DP) (training from scratch with noise-injection algorithms like DP-SGD). Both approaches are computationally prohibitive for large-scale models and often lead to performance degradation.

Figure 1: Comparison of the proposed Knowledge Unlearning approach against traditional retraining-based methods like Data Preprocessing and Differential Privacy.

Methodology: Knowledge Unlearning

This paper introduces "knowledge unlearning," a post-hoc method to surgically remove specific memories from a trained model. The core technique is Gradient Ascent. While standard training (Gradient Descent) minimizes the loss function to help the model predict the next token correctly, unlearning maximizes the loss for a specific target sequence. Essentially, the model is trained to become as poor as possible at predicting specific private strings.

Procedural Findings

  1. Sequential Unlearning: The researchers found that attempting to unlearn a large batch of data simultaneously leads to "catastrophic forgetting," where general model intelligence drops. However, unlearning small chunks of data sequentially allows the model to forget targets while maintaining its general capabilities.
  2. Efficiency: Unlike retraining, which can consume hundreds of A100 GPU days, unlearning requires only a few parameter updates, typically costing around ~0.001 A100 GPU days.

Experimental Setup

The experiments utilized GPT-Neo models (ranging from 125M to 2.7B parameters). The target data for unlearning consisted of 15,000 examples from the Training Data Extraction Challenge, spanning 16 domains of the Pile corpora (e.g., Enron Emails, GitHub, PubMed). To ensure the models remained functional, they were evaluated on 9 NLP classification tasks (e.g., Hellaswag, PIQA) and 4 dialogue tasks.

Key Results and Discussion

The study demonstrates that unlearning provides a stronger empirical privacy guarantee than data deduplication or DP-decoding when sensitive data is known beforehand.

Figure 6: Empirical results showing the impact of unlearning (UL and UL+) on Extraction Likelihood (EL), Memorization Accuracy (MA), and general language modeling performance across different model sizes.

Privacy and Performance

As shown in the results, GPT-Neo models reached the "forgetting threshold" within 5–17 epochs. For the 2.7B parameter model, the Extraction Likelihood (EL)—the probability of an adversary successfully extracting a sensitive suffix—dropped from over 70% to approximately 1.6% after unlearning, with negligible impact on general language modeling accuracy.

The "Unlearning Paradox"

Interestingly, unlearning certain data actually improved performance on benchmarks like Lambada. For the GPT-Neo 1.3B model, accuracy increased by over 10%. This suggests that removing "easy-to-extract" outliers may help the model generalize better. However, the success of unlearning is domain-sensitive; structured data like code is easier to forget without collateral damage than unstructured raw text.

Conclusion and Limitations

Knowledge unlearning offers a highly efficient, post-hoc solution for privacy mitigation in LMs. However, the method requires knowing the specific target sequences to be forgotten a priori. Future work is needed to investigate the "generalization of forgetting"—where unlearning one chunk of data makes the model faster at forgetting similar subsequent chunks—and to ensure that unlearning one user's data does not inadvertently affect the privacy or performance of others.

Summary updated

Abstract

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.
BibTeX
@inproceedings{jang2023knowledge,
  author = {Joel Jang and Dongkeun Yoon and Sohee Yang and Sungmin Cha and Moontae Lee and Lajanugen Logeswaran and Minjoon Seo},
  title = {{Knowledge Unlearning for Mitigating Privacy Risks in Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=TkVue8LoYe},
}
Figure