Abstract
Previous works have evaluated memorization by comparing model outputs with training corpora, examining how factors such as data duplication, model size, and prompt length influence memorization. However, analyzing these extensive training corpora is highly time-consuming. To address this challenge, this paper proposes an innovative approach named ROME that bypasses direct processing of the training data. Specifically, we select datasets categorized into three distinct types—context-independent, conventional, and factual—and redefine memorization as the ability to produce correct answers under these conditions. Our analysis then focuses on disparities between memorized and non-memorized samples by examining the logits and representations of generated texts. Experimental findings reveal that longer words are less likely to be memorized, higher confidence correlates with greater memorization, and representations of the same concepts are more similar across different contexts.
Introduction
Large Language Models (LLMs) act as massive repositories of information, soaking up trillions of tokens during their training phase. While the goal is for these models to acquire general knowledge, they frequently memorize specific text fragments verbatim. This phenomenon of memorization raises significant concerns regarding privacy, data leakage, and copyright. Traditionally, studying memorization required comparing model outputs against gargantuan training datasets—a process that is computationally exhaustive and often impossible for closed-source models where training data is proprietary.
This paper introduces the ROME framework, which proposes a clever workaround. Instead of looking at the training data, ROME focuses on the model's internal behavior—specifically its "confidence" (logits) and its internal "thoughts" (hidden representations)—to understand the underlying mechanics of memorization.
Methodology: The ROME Framework
The core challenge in memorization research is the "Needle in a Haystack" problem: proving a model has memorized something usually requires finding that exact string in a multi-terabyte training set. ROME redefines memorization as the model's ability to produce correct, fixed answers for specific types of "atomic" knowledge—information that must be learned as a unit to be understood.
The framework relies on three pillars:
- The "Block In Block Out" (BIBO) Characteristic: Identifying datasets where the meaning of a text segment is derived from its entire structure (e.g., idioms). Correct completion of these segments is a strong signal of memorization.
- Comparative Analysis: Splitting samples into memorized (correctly answered) and non-memorized (incorrect) groups to identify statistical differences.
- Multi-Level Inspection: Analyzing three layers of model output: surface-level text features, raw logits (probability scores), and internal representations (hidden vector states).
Experimental Setup and Datasets
The study utilizes a diverse range of datasets categorized by their memorization signatures:
- Context-Independent: IDIOM (850 English idioms) and TangPoetry (1,500 ancient Chinese poems). These represent "all-or-nothing" memorization tasks.
- Conventional: ProperNoun (rare and common nouns) and Terminology (medical terms).
- Factual: PopQA, LAMA-UHN, and CelebrityParent. The latter tests "reversal relations" (e.g., if the model knows a parent, does it know the child?).
Key Findings and Results
Experimental results across models like LLaMA, Gemma, and Mistral reveal several distinct signatures of memorized content:
The Confidence Signature: Memorized samples exhibit a massive gap in probability scores compared to non-memorized ones. For instance, in the PopQA dataset, LLaMA-2 13B showed an average probability of 0.8362 for memorized instances versus only 0.5386 for non-memorized ones. This suggests that memorization is fundamentally linked to high-certainty internal states.
Complexity and Length: There is a clear trend showing that longer words are less likely to be memorized. The increased complexity of longer character strings makes them harder for the model to "lock in" during training. Conversely, for context-independent data like poetry, longer prompts significantly increase the likelihood of recall by providing more "hooks" for the model to access the stored block.
Internal Geometry and Stability: Using Principal Component Analysis (PCA), researchers found that memorized and non-memorized samples occupy different "neighborhoods" in the model's internal vector space. Furthermore, for memorized facts, the model's internal representation of a concept remains remarkably stable even when the surrounding prompt changes, suggesting that memorized knowledge is stored as a context-independent "block."
Conclusion
ROME provides a novel lens for viewing LLM memorization without the need for original training corpora. By identifying the "geometric" and probabilistic signatures of memorized text, we can better understand how models store and recall information. While the study confirms phenomena like the "Reversal Curse"—where models memorize sequences in one direction but struggle with the reverse—it also opens doors for future work in developing more efficient "unlearning" algorithms and data leakage filters based on internal model states.
Summary updated