Skewed Memorization in Large Language Models: Quantification and Decomposition

Hao Li, Di Huang, Ziyu Wang, Amir M. RahmaniarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) are known to memorize portions of their training data, posing significant privacy and copyright risks. However, this memorization is not uniform. This paper argues that focusing on average-case memorization rates is insufficient, as it overlooks the "skewed" nature of the phenomenon: a tiny fraction of "worst-case" samples are memorized much more intensely than the rest of the dataset. This research introduces a framework to quantify this skew and understand the underlying drivers of memorization in realistic Supervised Fine-Tuning (SFT) settings.

The Problem: Beyond Average-Case Metrics

Traditional assessments of memorization often rely on mean rates across a dataset. This approach is akin to evaluating bank security by the average amount stolen daily, ignoring rare but catastrophic "heists." In realistic SFT settings, most data is generalized, but a small subset is memorized perfectly. This study seeks to analyze why certain samples are more "sticky" and how to quantify this risk accurately without relying on unrealistic setups like massive repetition.

Methodology

The researchers propose a prefix continuation framework to measure memorization. By providing the first 100 characters of a training sample as a prompt (the prefix), they measure the match length ($n_{pre}$) of the model's continuation.

To analyze the distribution of these match lengths, the authors employ:

  • Non-parametric Sampling: Resampling techniques to estimate the "tail" of the distribution (worst-case risks) without making assumptions about the underlying data distribution.
  • Probabilistic Decomposition: Breaking down sequence memorization into token-by-token probabilities to test for "contextual stickiness."
  • Embedding Similarity Gap ($\Delta S$): A metric quantifying "local data density" by comparing the similarity of a sample's prefix to other data versus the similarity of the entire sequence.

Key Insights: The Mechanics of Memorization

The "Loss" Mirage and Dataset Size

A common misconception is that high training loss implies low memorization. The authors demonstrate that extreme memorization cases can emerge very early in training, even while overall loss remains high. Furthermore, dataset size acts as an accelerator; smaller datasets reach maximum memorization lengths much faster than larger ones, often hitting peak memorization within just 100 training steps.

Figure 3: Training dynamics showing the relationship between loss, training steps, and maximum word memory across different dataset sizes.

Contextual Stickiness and Data Density

Memorization is not just an inherent property of a data point but is heavily influenced by its "neighborhood" in the embedding space. The "One-Way Street" hypothesis suggests that memorization is most likely when a common prefix (high input similarity) consistently leads to a single specific ending. Conversely, if a prefix appears in many contexts with different endings (high entropy), the model is less likely to memorize any single instance.

Results: Quantifying the Skew

The study reveals significant "quantile shifts" in memorization. In a mixed dataset environment, the maximum memorization length can exceed 40 tokens as early as the first checkpoint (epoch 10), while the mean remains significantly lower. This divergence highlights the necessity of looking at the tail of the distribution rather than just the average.

Figure 4: Variables vs Step for the mixed dataset, illustrating the divergence between mean memorization and high-percentile (skewed) cases.

Evaluations also showed that small-scale sampling systematically underestimates risk. For instance, a sample size of 455 might estimate a maximum memorization of 8 tokens, whereas the true maximum in the full set is double that. There is also a clear positive correlation between a sample's similarity to other training points and its likelihood of being memorized.

Conclusion and Future Directions

While this work provides robust tools for detecting and quantifying skewed lexical memorization, several questions remain. Future research should distinguish between verbatim lexical memorization and "factual" memorization, where information is remembered but paraphrased. Additionally, investigating "embedding collapse"—where internal representations lose diversity during training—may provide deeper insights into the drivers of memorization intensity and help develop better mitigation strategies.

Summary updated

Abstract

Memorization in Large Language Models (LLMs) poses privacy and security risks, as models may unintentionally reproduce sensitive or copyrighted data. Existing analyses focus on average-case scenarios, often neglecting the highly skewed distribution of memorization. This paper examines memorization in LLM supervised fine-tuning (SFT), exploring its relationships with training duration, dataset size, and inter-sample similarity. By analyzing memorization probabilities over sequence lengths, we link this skewness to the token generation process, offering insights for estimating memorization and comparing it to established metrics. Through theoretical analysis and empirical evaluation, we provide a comprehensive understanding of memorization behaviors and propose strategies to detect and mitigate risks, contributing to more privacy-preserving LLMs.
BibTeX
@misc{li2025skewed,
  author = {Hao Li and Di Huang and Ziyu Wang and Amir M. Rahmani},
  title = {{Skewed Memorization in Large Language Models: Quantification and Decomposition}},
  year = {2025},
  eprint = {2502.01187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01187},
}
Figure