Associative Priming as a Signature of Stubborn Reasoning Errors

Some wrong answers from a language model are "stubborn": the model gives the same incorrect answer on every sample, so tools that flag uncertainty by looking for disagreement among samples see a confident model and pass the error through. We propose a cheap test borrowed from the psychology of word recognition. Feed the model only the loose content words of a question, with the structure that makes it a problem scrambled away, and check how much this raises the probability of the model's answer. If the scrambled words alone pull the answer up about as much as the real question does, the answer is coming from word association rather than from working the problem. We predict that stubborn errors look like this, that correct answers do not, and that the reasoning text a model writes for an association-driven answer is mostly a justification produced after the fact.

The test comes from the lexical decision task, in which people judge as fast as possible whether a letter string is a real word. A related "prime" word shown just before the target (DOCTOR before NURSE) speeds the response. A prime flashed too briefly to be consciously read ("masked priming") separates automatic association from deliberate processing. Cognitive Effects in Large Language Models showed that a model's probability for a target word works as a stand-in for human response speed: GPT-3 showed semantic priming on the Semantic Priming Project at $p<0.001$. We move that logic from single word pairs to multi-step questions, use it to catch stubborn errors, and track how it changes as models cross the size range where multi-step reasoning starts to work.

Core Claim

The claim has three parts. First, a model can commit to an answer early for two different reasons. It can be competent enough to compute the answer directly, or the answer can be lit up by association with surface words in the question. Second, only the associative kind predicts stubborn errors and reasoning text that does not causally drive the answer. Third, the two kinds can be told apart from output probabilities alone, with no gradients, by comparing three conditions:

  • the probability of the answer after the real question;
  • the probability after a "cue-only" prime that keeps the question's content words but destroys their relations;
  • the probability after a matched unrelated question, which controls for how common the answer is in general.

This matters for an open puzzle. On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models measured how early the answer can be read off a truncated reasoning trace. It found that stronger models such as GPT-4 decide earlier and are therefore scored as less faithful. Their measure cannot say whether that early decision is competence or association. Ours is built to say exactly that.

Let $s(c)=\log p(a\mid c)$ be the log-probability the model gives its own final answer $a$ after context $c$. Let $q$ be the real question, $\tilde q$ the cue-only prime, and $u$ a matched unrelated question with the same answer format. The associative share is

$$\rho = \frac{s(\tilde q) - s(u)}{s(q) - s(u)},$$

the fraction of the question's facilitation of the answer that survives scrambling. Values of $\rho$ near 1 mean association explains the answer. Values near 0 mean the answer depends on how the question's words are put together. We report $\rho$ only when the denominator exceeds a fixed margin, and otherwise fall back to the numerator alone.

Research Program

Building the primes. The work first builds cue-only primes for each benchmark. For commonsense and truthfulness items, the prime is the question's content words in shuffled order, with function words and negations removed. For arithmetic word problems, the prime is the entities and numbers listed without the relations between them. For abstract computation tasks, the prime is the symbol inventory without the rules.

Each prime gets two controls. A second shuffle checks that $\rho$ does not hinge on one particular ordering. A "structure-only" prime keeps the syntax but swaps in unrelated content words. Together these play the role of the related-versus-unrelated contrast in classic priming studies. Unrelated questions are matched to their targets on answer type and length.

Building the stubborn sets. Next we build stubborn subsets, following the construction in From Flat Facts to Sharp Hallucinations. We draw 20 samples at temperature 0.7 and keep questions where at least 90% of samples agree. We then label each kept answer correct or incorrect. This leaves a population where disagreement-based detectors have nothing to work with by design. The detection task is to separate confident-wrong from confident-right.

Linking $\rho$ to faithfulness. We test whether high $\rho$ marks reasoning text that does not causally drive the answer. For each item we measure how much the final answer depends on its reasoning trace. We do this by corrupting steps and measuring the change in answer probability, in the spirit of the treatment-effect analysis in How Likely Do LLMs with CoT Mimic Human Reasoning? and the step-necessity scores of Can Aha Moments Be Fake?. The second paper reports a mean True Thinking Score of only 0.03 on AIME for Qwen-2.5. Since most steps are decorative on average, the useful question is which items have decorative traces. We predict $\rho$ answers it.

Scaling. Finally, we run everything across one model family at sizes from about 0.5B to 72B parameters. The literature reports reasoning floors between roughly 2B and 100B depending on the task, including a floor near 4B on TMBench (Computational Reasoning of Large Language Models). Our prediction is specific. Below the floor nearly every answer, right or wrong, is associative, so $\rho$ is high everywhere and cannot discriminate. Above the floor, correct answers become structure-dependent and $\rho$ starts to separate them from errors. At the largest scales, correct answers are decided early but with low $\rho$. That last pattern would recast the "GPT-4 paradox" as competent early commitment, not rationalization.

How the input papers fit. Each direction plays a different role:

  • Priming supplies the method.
  • Consistent hallucination is the target population. RELIC attributes it to models favouring frequent associations, which motivates the frequency control $u$.
  • The reasoning threshold is the axis along which the diagnostic should switch on.
  • Post-hoc rationalization is the downstream property $\rho$ should predict.

Unveiling Confirmation Bias in Chain-of-Thought Reasoning found that low entropy on the direct answer predicts rationalizing traces in 7B–8B models. That finding motivates us but cannot separate the two kinds of confidence. We aim to split its low-entropy group.

Baselines

Naive reference: majority-vote agreement. The do-nothing reference is the agreement rate among samples, which equals semantic entropy restricted to our stubborn sets. By construction it should sit near chance there. On the SQuAD stubbornness subset with Llama-2-7B, From Flat Facts to Sharp Hallucinations reports semantic entropy at 0.58 area under the ROC curve (AUROC).

Strongest existing detector: EPGS. Embedding-Perturbed Gradient Sensitivity (EPGS) is the strongest detector for stubborn errors in the file. It perturbs input embeddings and measures gradient sensitivity, on the view that stubborn errors sit in sharp regions of the loss surface. The same paper reports 0.7373 AUROC for EPGS on that subset with Llama-2-7B. We run our method on the identical subset and model, and use the authors' perturbation settings for EPGS. Beating it means exceeding 0.7373 AUROC there while using only output probabilities.

EPGS has only been shown on factoids. On our multi-step sets we run it on the answer tokens with the same perturbation budget. There we expect it to weaken, though no reported result was found on Lacuna for EPGS on multi-step reasoning traces.

Most direct competitor: early-answering curves. The Ley et al. early-answering curve truncates the trace at each step, forces an answer, and scores the area over the resulting curve. We use it as a stubborn-error detector with the same models, items, and sampling budget. No reported result was found on Lacuna for early-answering curves used as a hallucination detector. We also test the combination of early answering with $\rho$, because the claim is that $\rho$ adds information early answering lacks.

Frequency-only ablation (PMI). This ablation isolates the key ingredient. It drops the cue-only prime and keeps only the frequency control, giving the standard pointwise mutual information (PMI) score $s(q)-s(u)$. A second ablation replaces $u$ with an empty context. If PMI matches $\rho$, the priming decomposition adds nothing beyond frequency correction.

Paraphrase consistency (SAC³). SAC³ checks agreement across meaning-preserving rewrites and other models. It is run with the same number of model calls as our three-prompt probe plus samples. No reported result was found on Lacuna for SAC³ on stubborn subsets.

Tuning budget. All methods get one shared threshold-free metric (AUROC) and the same held-out split. No method is tuned on test items.

Why not just…?

Why not just use early answering, which already shows when the model commits? This is the real threat. Early answering tells you that the answer was available before the reasoning. It does not tell you why, and the Ley et al. finding that capable models commit early implies early commitment is common among correct answers too. Our prediction is that early answering separates confident-right from confident-wrong poorly at large scale, where both commit early, while $\rho$ still separates them. If early answering alone matches $\rho$ at every scale, we will report that the associative decomposition is redundant.

Why not just subtract the answer's base rate (PMI)? Frequency overshadowing is one documented driver of stubborn errors, and PMI corrects for it directly. But TruthfulQA-style misconceptions and arithmetic slips such as "add the salient numbers" are tied to the question's words, not to the answer's overall frequency. A base-rate correction leaves that question-specific association untouched. The PMI ablation above settles this. If it ties $\rho$, we drop the cue-only prime and say so.

Why not just paraphrase the question and check consistency? Stubborn errors are by definition stable under resampling. Cognitive Effects in Large Language Models and the confirmation-bias results suggest they are anchored in the content words that paraphrases preserve. Our probe perturbs in the opposite direction: it destroys meaning and keeps the words. Stability under that perturbation is the warning sign. The head-to-head with SAC³ tests this.

Why not just use gradient sensitivity? EPGS needs gradients, so it does not apply to closed models, and it has only been shown on single-entity answers. It remains the bar to clear on its own benchmark. If it beats $\rho$ there, our contribution narrows to multi-step and output-probability-only settings.

Proposed Experiments

Detection. The first check is stubborn-error detection measured by AUROC. We use the SQuAD stubbornness subset for direct comparison with EPGS. We then add stubborn subsets we build from GSM8K, StrategyQA, CommonsenseQA, and TruthfulQA, across the size sweep. Success means $\rho$ beats PMI and early answering above the reasoning floor on the multi-step sets, and approaches or beats 0.7373 on the SQuAD subset.

Link to rationalization. The second check asks whether, among items with the same final answer, higher $\rho$ predicts a smaller causal effect of the reasoning trace on the answer. This would show the probe tracks rationalization, not just error.

Scale profile. The third check is the predicted profile across size. Discrimination should be near chance below the task's floor and rise above it. TMBench serves as a near association-free control, where the cue-only prime should carry almost no facilitation at any scale.

Intervention. The decisive causal check builds counter-associative variants of high-$\rho$ stubborn items: the same words, but a structure that changes the correct answer. If association drives these answers, high-$\rho$ items should keep their old answer under the variant far more often than low-$\rho$ items. A null result here would falsify the core claim.

As a secondary test, we apply the skeleton prompting of Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation to high-$\rho$ and low-$\rho$ items. This shows whether structural scaffolding repairs association-driven errors or just fills its tags with justification.