Understanding Verbatim Memorization in LLMs Through Circuit Discovery

Ilya Lasy, Peter Knees, Stefan WoltranACL 2025

TL;DR

This paper dives into the "black box" of Large Language Models (LLMs) to understand how they memorize and reproduce training data verbatim. By using a technique called "circuit discovery," the researchers pinpoint specific computational pathways within the model responsible for memorization, revealing that the ability to prevent memorization is more generalizable than the ability to induce it. This work offers a mechanistic understanding that could lead to better control over LLM memorization.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly powerful, but a significant challenge is their tendency to "memorize" and reproduce exact text from their training data. This verbatim memorization has both upsides (storing knowledge) and serious downsides (privacy risks, copyright issues, and making it hard to distinguish learned knowledge from copied text). While previous work has shown that memorization is distributed across LLM layers, the precise mechanisms remain unclear. This paper aims to pinpoint the specific computational "circuits" within LLMs that drive this memorization behavior, moving beyond correlation to causation.

Methods

The core of this research lies in circuit discovery, a mechanistic interpretability technique that aims to reverse-engineer LLM behavior by identifying minimal computational subgraphs (circuits) responsible for specific functions. The authors employ activation patching (specifically, Edge Attribution Patching with Integrated Gradients - EAP-IG) to identify these circuits. This involves creating carefully designed contrastive datasets where small changes can elicit different model behaviors (memorization vs. generation). By "patching" (intervening) in the model's computation, they can isolate the components crucial for memorization. They also introduce the concept of divergence points, the specific token positions where a model transitions from memorized output to novel generation.

Datasets

The primary dataset used is the Wikipedia subset of The Pile. To facilitate circuit discovery, the authors construct two novel contrastive datasets:

  • Memorization Decision Dataset: Designed to identify components that initiate memorization.
  • Branch Comparison Dataset: Designed to identify components that continue along a memorized path versus those that branch into novel generation.

These datasets are built by identifying highly memorized sentences and then creating "clean" (memorized) and "corrupted" (non-memorized but semantically similar) contexts around divergence points.

Key Insights and Ideas

  • Initiation vs. Maintenance: Circuits that initiate memorization can also maintain it, but circuits that only maintain memorization cannot trigger its start. This suggests a distinct "trigger" mechanism.
  • Generalizability Asymmetry: Mechanisms that prevent memorization are robust and transfer well across different text domains (like GitHub, Enron Emails, Common Crawl). In contrast, mechanisms that induce memorization are more context-dependent and specific to certain domains.
  • Circuit Compactness: The researchers found surprisingly small circuits (as little as 0.04% of model edges) responsible for key memorization behaviors, particularly in the Branch Comparison task.
  • Memorization Prevention is Easier: It's easier to disrupt or prevent memorization (noising approach) than to induce it in non-memorized contexts (denoising approach). This might be because memorization involves a "trigger" circuit that can be easily disrupted, alongside a more distributed storage system.

Key Results

  • Identified compact circuits (less than 1% of model edges) with high faithfulness (≥ 85%) for both Memorization Decision and Branch Comparison tasks.
  • The Branch Comparison circuit was remarkably minimal, requiring only 14 edges.
  • Memorization Decision circuits showed strong cross-task generalization, effectively preventing memorization when applied to the Branch Comparison task.
  • Cross-corpus experiments showed that memorization prevention mechanisms transferred effectively across GitHub, Enron Emails, and Common Crawl datasets, reducing memorization accuracy significantly (e.g., from 77.6% to 9.3% on GitHub for the Memorization Decision circuit).
  • Memorization induction circuits showed less cross-corpus transfer, suggesting context-dependency.

Limitations and Future Work

  • Attribution Patching Approximation: The study relied on EAP-IG, an approximation of direct activation patching. Future work should validate findings with direct activation patching.
  • Model Size: The experiments were conducted on a single, relatively small model (GPT-Neo-125m). Larger models might have more complex or differently distributed memorization mechanisms. Future work should explore scaling across model sizes and architectures.
  • Dataset Specificity: The contrastive datasets were constructed with specific criteria for memorization detection. While cross-corpus generalization was tested, it's possible these criteria don't capture all memorization phenomena. Future work should explore more diverse datasets and memorization criteria.
  • Generalizability of Circuits: While some circuits showed cross-task and cross-corpus generalization, the paper notes that small circuits might be artifacts or incomplete components. Future work should investigate how memorization mechanisms scale and generalize across diverse settings.

Summary updated

BibTeX
@inproceedings{lasy2025understanding,
  author = {Ilya Lasy and Peter Knees and Stefan Woltran},
  title = {{Understanding Verbatim Memorization in LLMs Through Circuit Discovery}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure