SoK: Memorization in General-Purpose Large Language Models

Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, Robert WestarXiv 2023 · OpenReview

Abstract

Large Language Models (LLMs) are advancing at a remarkable pace, with myriad applications under development. Unlike most earlier machine learning models, they are no longer built for one specific application but are designed to excel in a wide range of tasks. A major part of this success is due to their huge training datasets and the unprecedented number of model parameters, which allow them to memorize large amounts of information contained in the training data. This memorization goes beyond mere language, and encompasses information only present in a few documents. This is often desirable since it is necessary for performing tasks such as question answering, and therefore an important part of learning, but also brings a whole array of issues, from privacy and security to copyright and beyond. LLMs can memorize short secrets in the training data, but can also memorize concepts like facts or writing styles that can be expressed in text in many different ways. We propose a taxonomy for memorization in LLMs that covers verbatim text, facts, ideas and algorithms, writing styles, distributional properties, and alignment goals. We describe the implications of each type of memorization - both positive and negative - for model performance, privacy, security and confidentiality, copyright, and auditing, and ways to detect and prevent memorization. We further highlight the challenges that arise from the predominant way of defining memorization with respect to model behavior instead of model weights, due to LLM-specific phenomena such as reasoning capabilities or differences between decoding algorithms. Throughout the paper, we describe potential risks and opportunities arising from memorization in LLMs that we hope will motivate new research directions.

Introduction

As Large Language Models (LLMs) become increasingly integrated into daily technology, understanding how they "remember" information is critical. This "Systematization of Knowledge" (SoK) paper argues that memorization in LLMs goes far beyond simple verbatim "regurgitation." Instead, it is a multi-faceted phenomenon where models internalize concepts, logic, and styles. While memorizing facts is essential for utility (e.g., answering questions accurately), it simultaneously creates a complex tug-of-war between model performance, privacy, security, and copyright law.

A Taxonomy of Memorization

To move beyond the narrow focus on verbatim string matching, the authors propose a novel taxonomy consisting of six categories of memorization. Each category presents unique opportunities for model performance and risks for data integrity.

  1. Verbatim Text: Exact sequences of tokens (e.g., a specific quote or code snippet).
  2. Facts: Relational knowledge about the world (e.g., "A is a student of B").
  3. Ideas and Algorithms: Underlying logic, plot points, or code structures (e.g., the steps of Quicksort).
  4. Writing Styles: Linguistic patterns, formality levels, and author-specific quirks.
  5. Distributional Properties: Statistical metadata about the training set (e.g., demographic distributions).
  6. Alignment Goals: Behavioral guidelines instilled during fine-tuning (e.g., safety constraints).
Table 1: Taxonomy of memorization in LLMs and its implications across various domains.

Key Insights and Implications

The Extractability Gap

A crucial distinction exists between memorization (information stored in weights) and extractability (information that can be prompted out). Current metrics often underestimate memorization because they only measure what is easily extractable via standard prompts. If an attacker finds the right "trigger," hidden memorized data can be leaked.

Structural Memorization

Research into Transformer architectures suggests a functional split: Multi-Layer Perceptron (MLP) modules likely act as "key-value stores" for facts, while attention layers manage the relationships between these stored concepts. Furthermore, the authors argue that Reinforcement Learning from Human Feedback (RLHF) is essentially a process of forcing the model to memorize a specific "style" of safe and helpful behavior.

Memorization creates significant legal friction. While facts themselves cannot be copyrighted, their specific expression can. LLMs blur this line by memorizing the "idea" of a work (like a fictional character's traits) even without verbatim copying, potentially leading to non-verbatim copyright violations. In privacy, even if a model doesn't leak a name, memorizing "linkable" facts (e.g., a rare medical condition combined with a specific location) can allow for the de-anonymization of individuals.

Key Results and Risks

The mapping of memorization types to real-world domains reveals several critical findings:

  • Security: LLMs used as coding assistants have been shown to leak proprietary API keys and internal code structures memorized during training.
  • Performance: Memorization is a double-edged sword. It allows for "zero-shot" quoting but can cause "instruction-overriding," where a model ignores a user's prompt because it is too focused on a memorized sequence.
  • Quantification: Studies indicate that models like GPT-J memorize at least 1% of their training data in a way that is easily extractable.

Challenges and Future Work

The paper identifies several gaps in current research. Most studies focus on the pre-training phase, leaving a lack of data on how much sensitive user-submitted information is memorized during supervised fine-tuning (SFT) or RLHF. Additionally, current detection methods are often "prefix-specific," failing to detect memorization if an attacker only knows the middle or end of a document. Finally, there remains no clear mathematical demarcation between "good" generalization (learning a concept) and "bad" memorization (storing a specific instance), calling for new research into reasoning-based memorization.

Summary updated

Abstract

Large Language Models (LLMs) are advancing at a remarkable pace, with myriad applications under development. Unlike most earlier machine learning models, they are no longer built for one specific application but are designed to excel in a wide range of tasks. A major part of this success is due to their huge training datasets and the unprecedented number of model parameters, which allow them to memorize large amounts of information contained in the training data. This memorization goes beyond mere language, and encompasses information only present in a few documents. This is often desirable since it is necessary for performing tasks such as question answering, and therefore an important part of learning, but also brings a whole array of issues, from privacy and security to copyright and beyond. LLMs can memorize short secrets in the training data, but can also memorize concepts like facts or writing styles that can be expressed in text in many different ways. We propose a taxonomy for memorization in LLMs that covers verbatim text, facts, ideas and algorithms, writing styles, distributional properties, and alignment goals. We describe the implications of each type of memorization - both positive and negative - for model performance, privacy, security and confidentiality, copyright, and auditing, and ways to detect and prevent memorization. We further highlight the challenges that arise from the predominant way of defining memorization with respect to model behavior instead of model weights, due to LLM-specific phenomena such as reasoning capabilities or differences between decoding algorithms. Throughout the paper, we describe potential risks and opportunities arising from memorization in LLMs that we hope will motivate new research directions.
BibTeX
@misc{hartmann2023sok,
  author = {Valentin Hartmann and Anshuman Suri and Vincent Bindschaedler and David Evans and Shruti Tople and Robert West},
  title = {{SoK: Memorization in General-Purpose Large Language Models}},
  year = {2023},
  eprint = {2310.18362},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18362},
}
Figure