Abstract
Previous works have evaluated memorization by comparing model outputs with training corpora, examining how factors such as data duplication, model size, and prompt length influence memorization. However, analyzing these extensive training corpora is highly time-consuming. To address this challenge, this paper proposes an innovative approach named ROME that bypasses direct processing of the training data. Specifically, we select datasets categorized into three distinct types—context-independent, conventional, and factual—and redefine memorization as the ability to produce correct answers under these conditions. Our analysis then focuses on disparities between memorized and non-memorized samples by examining the logits and representations of generated texts. Experimental findings reveal that longer words are less likely to be memorized, higher confidence correlates with greater memorization, and representations of the same concepts are more similar across different contexts.
Introduction
Understanding how Large Language Models (LLMs) store and retrieve information is critical for improving their reliability and safety. Traditionally, memorization is assessed by comparing model outputs against massive training corpora. However, as models like LLaMA-2 are trained on trillions of tokens, this approach becomes computationally prohibitive and often impossible when training data is proprietary.
The ROME framework (Memorization Insights from Text, Logits, and Representation) addresses this "data bottleneck" by analyzing internal model states. Instead of external validation against training sets, ROME examines how models behave when retrieving "atomic" knowledge—information that must be retrieved as a single unit to be correct.
The ROME Framework
The core of ROME is the "block in block out" characteristic. Certain text segments, such as idioms or specific poems, function as cohesive units. If a model completes these sequences perfectly, it likely indicates memorization rather than creative inference.
The framework analyzes five LLMs (including LLaMA-2, Mistral, and Gemma) across three dimensions:
- Textual Analysis: Investigating how context and word length influence correct retrieval.
- Logit Analysis: Measuring model "confidence" via the probability distribution of predicted tokens.
- Representation Analysis: Visualizing internal hidden states using dimensionality reduction (PCA) to distinguish memorized from non-memorized knowledge.
Experimental Setup
The study utilizes three categories of datasets based on how "fixed" the information is:
- Context-Independent: IDIOM (850 English idioms) and TangPoetry (1,500 ancient Chinese poems).
- Conventional: ProperNoun (300 entities) and Terminology (200 disease terms).
- Factual: PopQA, LAMA-UHN, and CelebrityParent, testing relational facts.
Key Insights
The Complexity Penalty
A significant finding is that longer words are less likely to be memorized. The increased complexity of longer character sequences appears to make them harder for models to "lock in" during training compared to shorter, high-frequency tokens.
Confidence and Internal Separability
There is a substantial "confidence gap" between memorized and non-memorized samples. For instance, in the PopQA dataset, LLaMA-2 13B predicted memorized tokens with ~84% probability, versus ~54% for non-memorized ones. Furthermore, PCA visualizations show that memorized and non-memorized data points often form distinct clusters, suggesting the model processes memorized information in a fundamentally different internal "space."
The Reversal Curse
LLMs often fail the "Reversal Curse" (e.g., knowing "A is B's parent" but not "B is A's child"). ROME discovered that even when a model fails a reversed query, the internal representations of the related concepts remain highly similar across contexts. This suggests the model "knows" the entity but lacks the specific retrieval path.
Conclusion
ROME provides a scalable method for probing LLM memory without training data access. Findings indicate that memorization is a high-confidence state with unique internal signatures. Future work may leverage these insights to develop "unlearning" techniques, allowing for the surgical removal of sensitive memorized data without compromising general reasoning capabilities.
Summary updated