Large language models show a "reasoning floor": below some size, accuracy on multi-step problems stays near chance, then jumps once the model is large enough, even though ordinary training loss improves smoothly the whole time. Attention-based explanations read a transformer's attention weights, the per-token mixing coefficients each layer uses to decide which earlier tokens to pull information from, as a picture of what the model "looked at." Interest is growing in using such signals to monitor and steer reasoning: picking out valid proofs, compressing chains of thought, correcting drift mid-generation. Nearly all of that work uses models at or above the floor. This proposal asks what happens right at the floor. When a model is just too small to solve an N-step problem, does it fail because its attention never finds the right facts, or because it finds them and then cannot combine them? And in the second case, do attention-based explanations look correct while saying nothing about why the answer is wrong?
The core claim is testable and, as far as this file shows, untested. Near the floor, attention routing and causal use separate. Sub-threshold models attend to the right evidence surprisingly often, yet that evidence is not causally carried to the answer. As a result, attention explanations become least faithful exactly where people most want to use them for diagnosis, and attention-based steering helps only when the failure is in routing. We will test this on synthetic in-context multi-hop tasks. There, the floor can be moved for a single fixed model by changing task difficulty, so model size does not change at the same moment. We then check whether the pattern carries over to pretrained models on GSM8K and MATH-500.
Why this matters and what prior work leaves open
Several papers motivate the gap. Thinking Sparks! uses edge attribution patching to find attention heads that reasoning post-training installs. Ablating 5–10 of them drops DeepSeek-R1-Distill-Qwen-1.5B on AIME '24 from 30.0 to 26.6. It does not ask whether smaller or sub-threshold models lack these heads or have them in an unstable form. ATTNPO finds "Key-Focus Heads" that put attention on the essential derivation steps, and uses them for process supervision at 1.5B and 7B. It leaves unexplained why the smaller model gains more (+7.3 vs +2.9 points). Geometry of Reason treats attention matrices as token graphs and shows that spectral roughness (HFER, the high-frequency energy ratio) separates valid from invalid Lean proofs with Cohen's d up to 3.30. That separation falls to d = 0.78 on natural-language MATH, and the paper never varies model scale. All three take attention as a meaningful signal of reasoning. None asks whether that meaning survives near the capability boundary.
Two other lines constrain the design. On the scaling side, Finding the Minimal Parameter Budget for Implicit Reasoning shows that multi-hop deduction on synthetic knowledge graphs needs roughly 124 extra parameters per bit of graph search entropy. That gives a quantitative handle on where a given model's floor should sit, and we reuse it to set task difficulty. Wu et al. and Chi et al. argue that small models lack the circuitry to track state at all. On the intervention side, Manifold-Guided Attention Steering (MAGS) assumes the opposite. It treats reasoning failures as drift in models that have the capacity, and corrects selected heads by projecting out an "error subspace" (+5.2 points on MATH-500 for Llama-3.1-8B). These two positions conflict, and they predict different things below the floor. Our routing-versus-composition decomposition is built to settle that conflict rather than assume either side.
The general worry that attention weight is not causal importance is the standing objection to attention explanations. Most work answers it in the abstract. We make it a quantity that we measure as a function of distance from the reasoning floor.
Core Claim
Define a model's routing accuracy on hop k as the share of attention mass, from the position that must retrieve hop k, that lands on the tokens of the correct supporting fact. Define causal reliance on that fact as the drop in correct-answer log-probability when only that fact's tokens are corrupted (activation patching). The faithfulness gap is the disagreement between attention-based and patching-based rankings of the in-context facts, measured as one minus their Spearman correlation.
We predict three things. First, routing accuracy degrades smoothly with task entropy, while task accuracy falls off a cliff. Second, the faithfulness gap peaks just below the floor. Third, forcing attention onto the correct facts (an "oracle routing" intervention) recovers accuracy well above the floor but not below it. If these hold, sub-threshold failure is mainly a composition failure that attention cannot reveal, and attention-based monitoring and steering are reliable only on the far side of the floor. If instead oracle routing rescues sub-threshold models, the MAGS view is correct even at small scale, and attention steering becomes a credible way to get around the floor. Either outcome is informative.
Research Program
The main testbed is a synthetic in-context multi-hop task. Each prompt contains a shuffled set of relational facts written with random entity IDs, including distractor facts that share entities with the true chain. A query requires composing N facts. Because the facts sit in the context instead of in the weights, every hop has a concrete set of tokens whose attention and causal role we can measure. That is impossible in the parametric setup of the minimal-parameter-budget paper, but we keep its entropy accounting. Search entropy is controlled by N (two to eight hops, matching GSM8K's step range) and by the number of distractors per entity.
We train a ladder of decoder-only transformers from scratch on this task, roughly 10M to 1B parameters, at matched tokens per parameter. For each size we sweep task entropy. The 124-parameters-per-bit rule predicts where each model's floor should fall, which gives an independent check that we have located it. Because difficulty can move a fixed checkpoint across its own floor, the analysis does not mix up "smaller model" with "different training run."
For each model–difficulty cell we compute routing accuracy per hop and per head. We compute causal reliance by corrupting one fact at a time. Following Thinking Sparks!, we run edge attribution patching with integrated gradients to find which heads actually carry hop information. We also compute HFER and the Fiedler value from Geometry of Reason on the same attention graphs. This gives, at every point on the difficulty axis, a direct comparison of what attention says against what the model causally uses.
The intervention arm has three conditions. Oracle routing overwrites the attention pattern of the patching-identified hop heads so that all their mass falls on the correct fact. A MAGS-style projection is fit from paired correct and incorrect rollouts in the same cell. A matched random-head control applies the same overwrite to non-hop heads. Oracle routing is the key ablation. It is the most generous possible version of "fix the attention," so if it fails below the floor, no attention-steering method can be expected to succeed there.
The transfer arm uses released model pairs of different sizes trained on the same data, such as the 1.5B and 7B distilled reasoning models used by ATTNPO and Thinking Sparks!, plus Llama-3.1-8B as MAGS's reference model. These run on GSM8K (Cobbe et al.) and MATH-500, with problems split by number of solution steps. Natural problems have no clean "correct fact" tokens, so we use the problem-statement quantities each step consumes, as annotated from reference solutions, as the routing target. We expect the transfer signal to be noisier, in line with the d = 0.78 drop on informal math reported by Geometry of Reason.
Baselines
The do-nothing reference for the faithfulness question is a uniform-attention or position-only ranking of the in-context facts, such as recency. It sets the faithfulness gap that carries no information, which every attention-based ranking must beat. For the intervention question, the reference is the unmodified model's accuracy in each cell.
The strongest existing explanation method here is causal, not attention-based: activation patching and EAP-IG as used in Thinking Sparks!. We treat it as ground truth rather than as a competitor to beat. The question is how closely cheap attention readouts track it as a function of distance to the floor. To be fair, patching and attention are computed on exactly the same prompts and checkpoints, and patching gets its full corruption budget.
The strongest existing attention-derived validity signal is HFER from Geometry of Reason. It reports 85–96% valid/invalid accuracy and d up to 3.30 on MiniF2F, d = 0.78 on MATH, and a 4.4–6.6% Pass@1 gain from Best-of-16 reranking. We recompute HFER on our models with the paper's single-threshold classifier. Our contribution counts as new if we show that HFER's separation, and per-hop routing accuracy, shrink or flip sign just below the floor in a way that the original single-scale evaluation could not detect. For the transfer arm we compare against the reported d = 0.78 on MATH only as context, since our models differ.
The strongest attention-based steering method is MAGS, which reports +5.2 points on MATH-500 with Llama-3.1-8B. We reimplement it with the same rollout budget for fitting the error subspace in every cell, and tune its drift threshold on a held-out split per model size so that small models are not handicapped by settings chosen for 8B. Oracle routing serves as both the upper-bound baseline and the ablation that isolates the routing ingredient.
For the validity-detection comparison we include output-token Shannon entropy as in PREGU, which reports 35.2% on GSM-Hard with LLaMA-3-8B against 28.2% for Soft Reasoning. We also include a linear probe on the residual stream that decodes the intermediate (bridge) entity at each hop, trained on the same number of labeled examples per cell.
Each comparison has its own success criterion. For faithfulness, we need a statistically reliable peak in the faithfulness gap within one entropy bit below the floor across at least three model sizes. For intervention, we need a crossover in which oracle routing recovers at least half of the gap to ceiling above the floor and less than a fifth below it. For detection, we need attention signals to lose their advantage over output entropy and residual probes below the floor while keeping it above.
Why not just…?
Why not just measure per-step accuracy and invoke the compounding $P^N$ story? The account in which success equals per-step reliability raised to the number of steps already predicts a cliff from smooth per-step gains, and needs no interpretability at all. It is a real threat as an explanation of the accuracy curve. It cannot, however, say why per-step reliability is low, and that is exactly the routing-versus-composition distinction that decides whether steering can help. We measure behavioral per-step accuracy in every cell anyway. If routing accuracy and oracle-routing gains are fully predicted by it, we will report that attention adds nothing beyond $P^N$.
Why not just use residual-stream probes instead of attention? A linear probe that decodes the bridge entity directly tests whether the intermediate result was computed, and may diagnose failure better than attention does. This is the most serious alternative, and it is included as a baseline. The two signals answer different questions, though. A probe says whether the intermediate exists; attention says where it was fetched from. The predicted dissociation is "attention correct, probe negative" below the floor. That pattern is the composition-failure signature, and it can only be seen by having both. If probes dominate attention everywhere, that is itself the cautionary result for attention-based explanation.
Why not just skip attention and use patching everywhere? Patching is the ground truth here, but it costs one forward pass per corrupted fact per example. That is why practical monitors like HFER, Key-Focus Heads and MAGS read attention instead. The point of the study is to say when the cheap readout can stand in for the expensive one. A finding that it never can near the floor would be useful practical guidance.
Why not just use output entropy? PREGU shows that output entropy flags critical tokens, but it cannot locate which earlier fact went wrong. We still compare against it directly on validity detection. If it matches attention signals at every scale, attention monitoring offers no detection advantage near the floor, and we will say so.
Decisive Checks
The first check is whether the floor found on the synthetic ladder matches the location predicted by the 124-parameters-per-bit rule. Without that match, "distance to floor" has no anchor. The second is the faithfulness-gap curve across difficulty for each model size, compared with the uniform and recency references. The third, and the one that decides the claim, is the oracle-routing crossover. If forcing correct attention rescues sub-threshold models, the circuit-absence view of Wu et al. is wrong for this task family, and MAGS-style steering deserves testing below 2B. If it does not, attention-based steering and explanation should come with a stated scale condition. The fourth is whether the same pattern appears between the 1.5B and 7B distilled models on step-stratified GSM8K and MATH-500. That would also explain ATTNPO's larger gains at 1.5B as a routing effect, which would be consistent with routing improvements mattering most near the floor but would not settle it.
References
- Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
- ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
- Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
- Computational Reasoning of Large Language Models
- ModeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
- Manifold-Guided Attention Steering
- Training Verifiers to Solve Math Word Problems
- Partial Reasoning in Language Models: Search and Refinement Guided by Uncertainty