Disentangling Reasoning from Memorization in Large Language Models

This direction is merged into another direction: benchmark data contamination. Its last version stays readable and citable here.

Version 1 of 1199 papers209 conceptsfrozen 2026-10-04

When a Large Language Model (LLM) solves a complex logic puzzle, is it performing genuine deduction or simply retrieving a "puzzle-game stereotype" from its vast training corpus? This question sits at the heart of current AI evaluation. As models scale, their ability to memorize high-frequency patterns—LaTeX math derivations, common Sudoku configurations, or standard SQL schemas—can create an illusion of intelligence that vanishes when the problem deviates from the training distribution.

The Contamination Crisis

The most immediate hurdle in measuring reasoning is data contamination. Shuo Yang and colleagues demonstrated that even small models can "fake" the performance of GPT-4 if they have been exposed to rephrased versions of benchmark questions during training. This suggests that many "emergent" abilities might actually be sophisticated forms of in-context recall rather than new cognitive leaps, a view supported by Sheng Lu, who argues that emergence is often a combination of model memory and better prompt adherence.

To unmask this, Chunyuan Deng introduced the TS-Guessing protocol. By asking models to fill in "masked" parts of multiple-choice questions (like the incorrect distractor options), researchers found that models like GPT-4 could guess these non-logical artifacts with high accuracy. This proves the models aren't just reasoning about the correct answer; they are remembering the specific text of the test itself.

Breaking Patterns with Counterfactuals

To isolate pure logic, researchers are moving toward "counterfactual" and "anti-factual" benchmarks. The intuition is simple: if a model truly understands the rules of logic, it should be able to apply them even when the premises are physically impossible or contradict its training data.

François Roewer-Després and Jinyue Feng developed the ACCORD framework, which uses scenarios like "If cats were vegetarians..." to force models away from their internal knowledge bases. Similarly, Peter Clark and the team at AI2 created BaRDa, which uses counterfactual examples to disentangle factual accuracy from reasoning ability. When the "truth" of the world is flipped, models often "parrot" the standard factual answer rather than following the logical chain to the new, counterfactual conclusion.

This "anchoring" effect becomes even more pronounced as models grow. Elena Stringli found that larger models are paradoxically worse at redefinition tasks (e.g., "redefine $\pi$ as 100") because their stronger parametric memory overrides the new instructions.

Symbolic and Structural Stress Tests

Another way to bypass memorization is to strip tokens of their semantic meaning. Eric Todd and David Bau showed that transformers can learn algebraic reasoning in-context using "pure variables." By removing the pre-encoded knowledge in token embeddings, they could identify specific emergent mechanisms—like commutative copying—that represent genuine algorithmic processing.

However, this reasoning is often brittle. Jordan Meadows and Andre Freitas used symbolic frameworks to "perturb" mathematical derivations. They found that while models excel at familiar LaTeX patterns, their performance drops by up to 80 points when simple changes are made, such as swapping variable names for Greek letters or changing the order of an equation. This suggests that the model's "logic" is often tied to the visual or statistical structure of the training data rather than the underlying mathematical rules.

The Reasoning Gap

When models are forced into "long-tail" scenarios—problems that are logically identical to common ones but statistically rare—the "reasoning gap" becomes clear. Jingcong Liang demonstrated this with HardcoreLogic, showing that models that solve 9x9 Sudokus easily fail on 16x16 variants.

The failure mode is often a decay in "reasoning hops." Atharva Pandey and Amit Sharma introduced DeduCE, which treats reasoning as a deductive proof. They found that LLM accuracy doesn't just fail on long contexts, but specifically decays as the number of required logical steps increases. In standard benchmarks, this decay is hidden because the model can "jump" to the correct answer using memorized patterns. Only by using novel, multi-step problems can we see the true limits of a model's deductive consistency.

Papers in this version

Showing 60 of 199 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_disentangling_reasoning_from_memorizatio_24252ab6,
  title = {Disentangling Reasoning from Memorization in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-reasoning-from-memorization-in-large-language-models/rdr_24252ab63f2641fdadc1ba4aeab619ea},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_disentangling_reasoning_from_memorizatio_24252ab6_v1,
  title = {Disentangling Reasoning from Memorization in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-reasoning-from-memorization-in-large-language-models/rdr_24252ab63f2641fdadc1ba4aeab619ea/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}