Training Data Extraction From Pre-trained Language Models: A Survey

Shotaro IshiharaACL 2023

TL;DR

This paper provides the first comprehensive survey of training data extraction attacks against pre-trained language models (PLMs). It systematically categorizes existing attack and defense strategies, defines various forms of memorization, and highlights empirical findings, offering a crucial overview for understanding and mitigating privacy risks in the rapidly evolving field of PLMs.

What Problem Does This Paper Address?

As pre-trained language models (PLMs) become increasingly powerful and widely deployed, a significant security concern has emerged: the potential for malicious actors to extract sensitive training data from these models. This "training data extraction" poses a direct threat to data privacy, potentially exposing personal information, proprietary data, or copyrighted material. While the risk is clear, the landscape of attacks and defenses is fragmented across numerous studies. This paper aims to consolidate this knowledge, providing a structured overview of the problem, existing solutions, and future research directions, making it easier for researchers and practitioners to navigate this critical area.

Methods

The paper's primary contribution is a comprehensive survey and systematization of existing research. The authors reviewed over 100 papers, categorizing them based on several key aspects:

  • Taxonomy of Memorization: They present a taxonomy of different definitions of "memorization" in PLMs, ranging from verbatim memorization (exact string reproduction) to approximate and counterfactual memorization, highlighting the nuances and challenges in defining what constitutes data leakage.
  • Attack and Defense Systematization: The paper outlines the typical procedure for training data extraction attacks, which generally involves two steps: candidate generation (producing numerous texts from a PLM) and membership inference (determining if a specific data point was part of the training set). It then systematically categorizes defense strategies into three phases: pre-processing (e.g., data sanitization, deduplication), training (e.g., differential privacy, regularization), and post-processing (e.g., confidence masking, filtering).
  • Empirical Findings: The survey highlights key quantitative findings from studies, focusing on factors like model size, data duplication, and prompt length, and their impact on memorization.

Datasets

The paper does not introduce new datasets. Instead, it discusses research that utilizes existing large-scale datasets for training and evaluating PLMs. A notable dataset mentioned is The Pile, a large, diverse text dataset used for training various GPT-Neo models in the evaluation studies discussed. The paper also references studies that use private or domain-specific datasets (e.g., clinical records, contracts, source code) where privacy concerns are particularly acute.

Key Insights and Ideas

  • Memorization is Multifaceted: The paper emphasizes that "memorization" isn't a single phenomenon. Different definitions (verbatim, approximate, counterfactual) capture distinct types of data leakage, each with its own implications and detection challenges.
  • Larger Models Memorize More: Empirical evidence strongly suggests a near-perfect log-linear relationship between model size and the amount of memorization. This implies that as models grow, so does their propensity to retain and reproduce training data.
  • Data Duplication is a Key Factor: The presence of duplicate strings in the training data significantly increases the likelihood of memorization. This highlights the importance of data cleaning and deduplication as a defense strategy.
  • Prompt Length Matters: Longer prompts can lead to a higher amount of memorization, suggesting that controlling prompt length can be a simple yet effective defense mechanism.
  • Defense Landscape is Evolving: While various defense strategies exist, they often come with trade-offs, such as performance degradation (e.g., differential privacy) or computational overhead. The paper underscores that no single defense is a silver bullet.
  • "Is Memorization Always Evil?": The paper raises a critical question about whether all forms of memorization are inherently bad. It suggests a need to differentiate between the leakage of sensitive personal information and the retention of general knowledge or stylistic patterns, balancing security with model utility.

Key Results

The paper synthesizes results from various studies, with a particular focus on the quantitative findings of Carlini et al. (2023b):

  • Model Size: A ten-fold increase in model size led to a 19 percentage point increase in memorization. For example, a 1.3B GPT-Neo model memorized significantly more than a GPT-2 of the same size.
  • Data Duplication: A clear log-linear trend was observed between the number of duplicate strings in the training data and the amount of memorization.
  • Prompt Length: For a 6B model, memorization increased from 33% with 50-token prompts to 65% with 450-token prompts.
  • Verbatim vs. Approximate Memorization: Both verbatim and approximate memorization studies showed similar trends regarding model size and data duplication.

Limitations and Future Work

Limitations:

  • Focus on Autoregressive Models: The survey primarily focuses on autoregressive language models, with less emphasis on other model types like masked language models or word embeddings.
  • English Language Bias: The research reviewed predominantly focuses on English language models, and research on other languages is encouraged.
  • Rapidly Evolving Field: The survey is based on information as of April 2023, and the rapid pace of development in PLMs means some content may become outdated quickly.
  • Definition Ambiguity: The boundary between memorization and general knowledge remains ambiguous, particularly with approximate memorization definitions.

Future Work:

  • Handling Fuzziness: Further research is needed to address the fuzziness of string matching, especially for non-English languages, and to explore its application in areas like image generation memorization and plagiarism detection.
  • Evaluation Schemas: Developing robust quantitative evaluation sets and metrics that go beyond simple accuracy and consider realistic attack scenarios is crucial.
  • Differentiating Memorization Types: Distinguishing between the leakage of sensitive personal information and acceptable memorization is essential for developing targeted defenses.
  • Balancing Security and Performance: Future work should focus on finding optimal balances between security concerns and model performance, considering the specific application context.
  • Broader Model Types: Expanding the scope to include other PLM architectures and attack vectors is necessary.

Summary updated

BibTeX
@inproceedings{ishihara2023training,
  author = {Shotaro Ishihara},
  title = {{Training Data Extraction From Pre-trained Language Models: A Survey}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure