Reason to Rote: Rethinking Memorization in Reasoning

Y. Le Du, Philipp Mondorf, Silvia Casola, Yao Yao, Robert Litschko, Barbara Plank2025 · DOI

Abstract

Large language models readily memorize arbitrary training instances, such as label noise, yet they perform strikingly well on reasoning tasks. In this work, we investigate how language models memorize label noise, and why such memorization in many cases does not heavily affect generalizable reasoning capabilities. Using two controllable synthetic reasoning datasets with noisy labels, four-digit addition (FDA) and two-hop relational reasoning (THR), we discover a reliance of memorization on generalizable reasoning mechanisms: models continue to compute intermediate reasoning outputs even when retrieving memorized noisy labels, and intervening reasoning adversely affects memorization. We further show that memorization operates through distributed encoding, i.e., aggregating various inputs and intermediate results, rather than building a look-up mechanism from inputs to noisy labels. Moreover, our FDA case study reveals memorization occurs via outlier heuristics, where existing neuron activation patterns are slightly shifted to fit noisy labels. Together, our findings suggest that memorization of label noise in language models builds on, rather than overrides, the underlying reasoning mechanisms, shedding light on the intriguing phenomenon of benign memorization.

Introduction: The Paradox of Benign Memorization

Deep learning models possess a dual nature: they are exceptional at learning generalizable rules while simultaneously being world-class at rote memorization. This creates a scientific puzzle known as benign memorization. If a model is trained on a dataset where a portion of the data contains incorrect "noisy" labels (e.g., "42 + 58 = 137"), why does this noise fail to degrade the model’s ability to solve clean problems at test time?

This research investigates the internal mechanics of this phenomenon. Rather than memorization overriding reasoning, the authors find that Large Language Models (LLMs) actually build their memories on top of their existing reasoning circuits.

Experimental Setup: FDA and THR

To isolate reasoning from simple pattern matching, the study utilized two synthetic tasks:

  1. Four-Digit Addition (FDA): A standard arithmetic task where the model predicts the sum of two numbers.
  2. Two-Hop Relational Reasoning (THR): A logic task requiring the model to identify a "bridge entity" to reach a final answer (e.g., "Who is the crush of the mentor of Adam?").
FDA task setup with clean and noisy labels.

In both datasets, label noise was introduced at a rate of $k=5\%$. This forced the model to memorize specific incorrect instances to achieve low training error, allowing researchers to observe the competition between generalization and rote learning.

Methodology

The researchers applied several mechanistic interpretability techniques to peek into the model's internal states:

  • Logit Lens & Linear Probing: Techniques to "read" the model's internal representations at each hidden layer.
  • Causal Interventions: Activation patching was used to swap internal states between "clean" and "noisy" runs to identify essential circuits.
  • Edge Attribution Patching (EAP): Identified specific sub-graphs of neurons (circuits) responsible for tasks.
  • Iterative Null-space Projection (INLP): Surgically removed specific information to test causal dependencies.

Key Insights and Results

1. Generalization Precedes Memorization

Training dynamics reveal that models initially learn the correct underlying logic for all samples, including the noisy ones. Only later in training does the model begin to "overwrite" these logical predictions with memorized noisy labels.

Learning dynamics and internal probing results.

2. The "Selective Outputting" Hypothesis

A major discovery was that even after a model has memorized a wrong answer, its middle layers often still compute the correct result. As shown in Figure 4(b), the Logit Lens revealed that at layer 2 of the FDA task, the model was ~50% accurate at predicting the correct sum for a noisy input, even though the final layer reached 100% accuracy for the memorized label. Memorization appears to happen in the final layers by selectively overriding the logical result.

3. Memorization as a Distributed Process

Memorization is not a simple "if-then" lookup table. It is a distributed process that aggregates information across multiple input tokens and intermediate reasoning steps. In the THR task, removing the "bridge entity" (the intermediate logical step) significantly harmed the model's ability to retrieve the memorized answer, proving that the memory is causally tied to the logical path.

4. Outlier Heuristics

In the addition task, the researchers identified "outlier heuristics." Neurons responsible for general math patterns slightly shift their activation levels to accommodate noisy labels. The model essentially "suppresses" the correct logical activation just enough to allow the memorized answer to surface for specific cases.

Conclusion and Future Directions

The findings suggest that memorization of label noise in LLMs builds on, rather than replaces, underlying reasoning mechanisms. This explains why reasoning capabilities remain robust despite exposure to noisy data.

However, the study notes limitations regarding model scale (4–8 layer Transformers) and the synthetic nature of the noise. Future work should explore how massive pre-training influences the memorization of new, noisy information and how this interacts with explicit "Chain-of-Thought" reasoning.

Summary updated

Abstract

Large language models readily memorize arbitrary training instances, such as label noise, yet they perform strikingly well on reasoning tasks.In this work, we investigate how language models memorize label noise, and why such memorization in many cases does not heavily affect generalizable reasoning capabilities.Using two controllable synthetic reasoning datasets with noisy labels, four-digit addition (FDA) and two-hop relational reasoning (THR), we discover a reliance of memorization on generalizable reasoning mechanisms: models continue to compute intermediate reasoning outputs even when retrieving memorized noisy labels, and intervening reasoning adversely affects memorization.We further show that memorization operates through distributed encoding, i.e., aggregating various inputs and intermediate results, rather than building a look-up mechanism from inputs to noisy labels.Moreover, our FDA case study reveals memorization occurs via outlier heuristics, where existing neuron activation patterns are slightly shifted to fit noisy labels.Together, our findings suggest that memorization of label noise in language models builds on, rather than overrides, the underlying reasoning mechanisms, shedding lights on the intriguing phenomenon of benign memorization. 1
BibTeX
@misc{du2025reason,
  author = {Y. Le Du and Philipp Mondorf and Silvia Casola and Yao Yao and Robert Litschko and Barbara Plank},
  title = {{Reason to Rote: Rethinking Memorization in Reasoning}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.437},
}

Loading related content...

Figure