TL;DR
Researchers have developed a "task-centric" framework to peek under the hood of LLMs, distinguishing whether a model is truly reasoning or just retrieving memorized facts. By analyzing internal signals like attention entropy and activation sparsity, they found that knowledge-heavy and reasoning-heavy tasks leave distinct "internal footprints," and that incorrect answers are characterized by significantly higher internal "noise" or variability during generation.
What Problem Does This Paper Address?
We currently evaluate LLMs using aggregate accuracy scores on massive benchmarks. However, a high score on a physics test doesn't tell us how the model got there: did it reason through the first principles of motion, or did it simply recognize a problem it saw during training?
This "black box" problem makes it difficult to identify a model's true architectural strengths and weaknesses. Previous attempts to disentangle knowledge from reasoning often relied on analyzing the model's text output (like Chain-of-Thought) or perturbing inputs. This paper argues we need a more fine-grained, internal approach to see how the model’s "brain" actually processes different types of cognitive demands.
Methods
The authors propose a framework that combines several internal diagnostic signals:
- Static Profiling: Measuring attention entropy (how focused vs. diffuse the model's attention is) and activation sparsity (what percentage of neurons are "firing") across all layers before the model starts generating an answer.
- Decoding-Time Trajectories: Tracking how these signals evolve token-by-token during the generation process.
- Layer-Wise Probing: Training simple linear classifiers on the model's internal hidden states to see if the correct answer is "linearly recoverable" (i.e., already present in the representation) at different depths.
- Representation Similarity: Measuring how distinctly the model represents a question when paired with a correct answer versus an incorrect distractor.
- NOTA (None of the Above) Stress Tests: A behavioral test where the ground-truth option is removed to see if the model can still reason its way to a "none of the above" conclusion, or if it collapses because it was relying on "parametric shortcuts" (matching memorized patterns).
Datasets
The study evaluates 16 LLMs (including the Qwen, Llama, Gemma, and OLMo families) across 24 diverse benchmarks. These are categorized into:
- Knowledge-Centric: Tasks like BioMixQA, SciQ, and MusicTrivia that primarily require factual retrieval.
- Reasoning-Centric: Tasks like LogiQA, MathQA, GSM8K, and GPQA-Diamond that require multi-step inference, calculation, or state tracking.
Key Insights and Ideas
- Internal Footprints: Knowledge-centric and reasoning-centric tasks elicit systematically different internal patterns. As shown in the layer-wise profiles, models process these tasks with varying levels of "focus" (attention entropy) and "efficiency" (activation sparsity).
- The "Noise" of Failure: One of the most striking findings is that incorrect answers are noisier. While the average level of attention might look similar for correct and incorrect answers, incorrect generations show much higher within-trajectory variability. Essentially, when a model is "confused" or wrong, its internal attention mechanism fluctuates wildly compared to the stable "consolidation" seen during a correct answer.
- Knowledge is "There," Reasoning is "Built": Probing reveals that for knowledge tasks, the correct answer is often linearly recoverable from the hidden states in the middle-to-late layers before generation starts. For reasoning tasks, this recoverability is much lower, suggesting the "answer" isn't just sitting there to be retrieved; it must be constructed through the decoding process.
Key Results
- CoT's Selective Benefit: Chain-of-Thought (CoT) prompting significantly improves reasoning scores (median +0.036) but has almost zero impact on knowledge retrieval scores (median +0.001).
- Scaling Disparity: Increasing model size leads to consistent gains in knowledge retrieval, but gains in reasoning are less monotonic and show diminishing returns at the largest scales.
- The Shortcut Trap: The NOTA stress tests revealed that many models—even with CoT—remain heavily dependent on the presence of the original correct option. When that option is removed, performance often collapses, indicating that the model was using a "shortcut" rather than robust, option-invariant reasoning.
- Input Sensitivity: The authors found that lower-layer activations are highly sensitive to how a prompt ends (e.g., adding "\nAnswer:"), warning researchers that "static" snapshots of model state can be easily confounded by prompt formatting.
Limitations and Future Work
- Language Scope: The study is limited to English benchmarks. Cross-lingual differences in tokenization and training data might yield different internal "footprints."
- Multiple-Choice Focus: Several key analyses (probing, NOTA) rely on the fixed candidate sets of multiple-choice questions. Tracing reasoning in purely open-ended generation remains a more complex challenge.
- Correlation vs. Causality: While the paper identifies strong associations between internal states and task types, it stops short of claiming these internal patterns cause the observed behavior, suggesting that future work using activation patching or causal tracing is needed.
Summary updated