Tracing Knowledge and Reasoning in LLM

Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin

TL;DR

Researchers have developed a "task-centric" framework to peek under the hood of LLMs, distinguishing whether a model is truly reasoning or just retrieving memorized facts. By analyzing internal signals like attention entropy and activation sparsity, they found that knowledge-heavy and reasoning-heavy tasks leave distinct "internal footprints," and that incorrect answers are characterized by significantly higher internal "noise" or variability during generation.

What Problem Does This Paper Address?

We currently evaluate LLMs using aggregate accuracy scores on massive benchmarks. However, a high score on a physics test doesn't tell us how the model got there: did it reason through the first principles of motion, or did it simply recognize a problem it saw during training?

This "black box" problem makes it difficult to identify a model's true architectural strengths and weaknesses. Previous attempts to disentangle knowledge from reasoning often relied on analyzing the model's text output (like Chain-of-Thought) or perturbing inputs. This paper argues we need a more fine-grained, internal approach to see how the model’s "brain" actually processes different types of cognitive demands.

Methods

The authors propose a framework that combines several internal diagnostic signals:

  1. Static Profiling: Measuring attention entropy (how focused vs. diffuse the model's attention is) and activation sparsity (what percentage of neurons are "firing") across all layers before the model starts generating an answer.
  2. Decoding-Time Trajectories: Tracking how these signals evolve token-by-token during the generation process.
  3. Layer-Wise Probing: Training simple linear classifiers on the model's internal hidden states to see if the correct answer is "linearly recoverable" (i.e., already present in the representation) at different depths.
  4. Representation Similarity: Measuring how distinctly the model represents a question when paired with a correct answer versus an incorrect distractor.
  5. NOTA (None of the Above) Stress Tests: A behavioral test where the ground-truth option is removed to see if the model can still reason its way to a "none of the above" conclusion, or if it collapses because it was relying on "parametric shortcuts" (matching memorized patterns).

Datasets

The study evaluates 16 LLMs (including the Qwen, Llama, Gemma, and OLMo families) across 24 diverse benchmarks. These are categorized into:

  • Knowledge-Centric: Tasks like BioMixQA, SciQ, and MusicTrivia that primarily require factual retrieval.
  • Reasoning-Centric: Tasks like LogiQA, MathQA, GSM8K, and GPQA-Diamond that require multi-step inference, calculation, or state tracking.

Key Insights and Ideas

  • Internal Footprints: Knowledge-centric and reasoning-centric tasks elicit systematically different internal patterns. As shown in the layer-wise profiles, models process these tasks with varying levels of "focus" (attention entropy) and "efficiency" (activation sparsity).
Figure 1: Layer-wise static internal profiles of Qwen2.5-7B-Instruct across 24 benchmarks. The heatmaps show attention entropy (left), where higher values indicate more diffuse attention, and activation sparsity (right), defined as the proportion of near-zero feed-forward activations. Benchmarks are ordered by the first principal component of the activation sparsity matrix to expose cross-benchmark variation in their layer-wise profiles.
  • The "Noise" of Failure: One of the most striking findings is that incorrect answers are noisier. While the average level of attention might look similar for correct and incorrect answers, incorrect generations show much higher within-trajectory variability. Essentially, when a model is "confused" or wrong, its internal attention mechanism fluctuates wildly compared to the stable "consolidation" seen during a correct answer.
Figure 3: Within-trajectory variability of attention entropy for Qwen2.5-7B-Instruct on SciQ, MathQA, and LogiQA. For each generation, the standard deviation of attention entropy is computed over the 101 aligned generated-token positions at the lower, middle, and upper depths (layers 3, 14, and 22).
  • Knowledge is "There," Reasoning is "Built": Probing reveals that for knowledge tasks, the correct answer is often linearly recoverable from the hidden states in the middle-to-late layers before generation starts. For reasoning tasks, this recoverability is much lower, suggesting the "answer" isn't just sitting there to be retrieved; it must be constructed through the decoding process.

Key Results

  • CoT's Selective Benefit: Chain-of-Thought (CoT) prompting significantly improves reasoning scores (median +0.036) but has almost zero impact on knowledge retrieval scores (median +0.001).
  • Scaling Disparity: Increasing model size leads to consistent gains in knowledge retrieval, but gains in reasoning are less monotonic and show diminishing returns at the largest scales.
  • The Shortcut Trap: The NOTA stress tests revealed that many models—even with CoT—remain heavily dependent on the presence of the original correct option. When that option is removed, performance often collapses, indicating that the model was using a "shortcut" rather than robust, option-invariant reasoning.
  • Input Sensitivity: The authors found that lower-layer activations are highly sensitive to how a prompt ends (e.g., adding "\nAnswer:"), warning researchers that "static" snapshots of model state can be easily confounded by prompt formatting.
Figure 2: Controls for the lower-layer activation sparsity in Figure 1 (Qwen2.5-7B-Instruct; mean over layers 0-3 at the final input token). Left and middle: benchmark means against mean prompt length and against the share of prompt tokens that contain a digit or mathematical symbol. Right: benchmark means under two controls against the original values. Length-adjusted values are predicted at a fixed length from a within-benchmark regression on log length; same-final-token values append “\nAnswer:” to every prompt. The dashed line marks equality.

Limitations and Future Work

  • Language Scope: The study is limited to English benchmarks. Cross-lingual differences in tokenization and training data might yield different internal "footprints."
  • Multiple-Choice Focus: Several key analyses (probing, NOTA) rely on the fixed candidate sets of multiple-choice questions. Tracing reasoning in purely open-ended generation remains a more complex challenge.
  • Correlation vs. Causality: While the paper identifies strong associations between internal states and task types, it stops short of claiming these internal patterns cause the observed behavior, suggesting that future work using activation patching or causal tracing is needed.

Summary updated

Abstract

Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.
BibTeX
@misc{wang2026tracing,
  author = {Yang Wang and Xiao Wang and Yiqi Liu and Chenghao Xiao and Eleni Vasilaki and Goran Nenadic and Chenghua Lin},
  title = {{Tracing Knowledge and Reasoning in LLM}},
  year = {2026},
  eprint = {2611.17461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17461},
}
Figure