Asking a language model to "think step by step" before answering, known as chain-of-thought (CoT) prompting, makes it more accurate. It also makes its mistakes harder to catch. A recent large study found that under CoT, the model's own confidence signals stop separating right answers from wrong ones. Those signals include how spread out its next-word probabilities are, the patterns in its hidden activations, and how sure it says it is. The loss showed up in the large majority of the configurations tested (Chain-of-Thought Prompting Obscures Hallucination Cues). That study measured everything on public benchmarks such as TriviaQA, PopQA, StrategyQA and MMLU. Models have very likely seen these benchmarks during training. We propose that part of the blind spot comes from that exposure. When a model reasons step by step over a question it has half-memorized, it can produce a fluent, low-uncertainty chain toward a wrong remembered answer. On a truly new question, a wrong chain may still show the usual signs of doubt.
We test this by building matched "twin" questions. Each twin copies the reasoning structure and entity popularity of a public benchmark item but uses facts the model cannot have memorized. We also contaminate a model on purpose with a known subset of those twins. People working together in an evidence-backed annotation interface write the twins and mark the first wrong step in each model reasoning chain. That step-level human label lets us ask where, inside a chain, the confidence signal comes apart from correctness. If the effect exists, it changes how hallucination detectors for reasoning models should be evaluated and trained. If it does not, the CoT blind spot is a real property of reasoning, and fixing it becomes more urgent.
Core Claim
We make two claims that experiments can settle. The first is an interaction claim. The drop in hallucination-detection quality (measured as AUROC, the probability that a detector ranks a wrong answer as more suspicious than a right one) caused by switching from direct answering to CoT is larger on items the model has been exposed to than on structurally matched fresh items. The second is a localization claim. On fresh items, the detection signal that CoT washes out at the final answer is still present at the first erroneous step of the chain. A detector that reads model internals at human-localized first-error steps therefore recovers much of the lost AUROC, and it does so better on fresh than on contaminated items.
What Prior Work Establishes and Leaves Open
The motivating result is Chain-of-Thought Prompting Obscures Hallucination Cues. It pairs four detector families (consistency, internal-state, self-evaluation, hybrid) with three CoT strategies on Llama-3.1-8B/70B, Mistral-7B and DeepSeek-R1-Distill-Llama-8B. AUROC fell in 465 of 768 configurations, calibration error rose, and consistency-based detectors held up best. It never asks whether its benchmarks were in training data, and we make that question the main variable.
Two findings constrain the design. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that membership detectors cannot be trusted to sort items into contaminated and clean for reasoning models. LiRA reaches 89.1% AUROC after contaminated fine-tuning, but it drops by roughly 14 points after one epoch of clean reinforcement learning with GRPO (a policy-gradient method), and reference-free detectors fall to about 50%. When a model is contaminated with full CoT traces, nearly all ten detectors land at 50–58%. We therefore cannot label contamination after the fact. We have to construct freshness and inject contamination ourselves, using that paper's Stage I and Stage II protocols. Separately, Snoopy shows that few-shot accuracy tracks how often terms appear in pretraining data. So fresh twins must be matched on entity popularity. Otherwise "fresh" simply means "obscure," and that alone would change uncertainty.
The dynamic-benchmark line supplies the idea of freshness by construction but not the domain. DyVal and BeyondBench generate problems procedurally and check them with solvers. Models strong on static benchmarks collapse on them, for example "nearly zero" for phi-1.5 and WizardMath on DyVal. These generators cover only math, logic and algorithmic tasks, so they cannot produce factual multi-hop questions, which is where the CoT blind spot was observed. LiveBench sources fresh questions from recent material, but it scores only final answers, and it shows that LLM judges fail on problems they cannot solve themselves. MRKE edits knowledge triples to create unseen multi-hop chains. We borrow that technique as one of our twin-generation modes.
The annotation method comes from two sources. CoTEVer turns human work from writing explanations into verifying them. Annotators rate decomposed CoT steps next to retrieved evidence and edit ungrounded ones. On 300 StrategyQA failures, "Wrong Fact" was the most common error at 43%. The authors admit that the tool handles factual errors but misses logical non-sequiturs, and they do not address the fact that StrategyQA is public. Let's Verify Step by Step shows that step-level human labels are valuable. Its process-supervised reward model reaches 78.2% on a MATH subset. It also shows that surfacing "convincing wrong-answer" solutions makes labeling 2.6 times more efficient. We use both ideas, adding a separate logical-error label and targeting a new question. We are not training a reward model. We are measuring how detectors behave at the step where the chain first fails.
Research Program
The first component is a twin corpus. We start from items in StrategyQA (implicit multi-hop yes/no), PopQA and TriviaQA (entity facts), and HaluEval, all used in the motivating study. For each seed item, annotators produce a twin in one of two ways. The first uses post-cutoff facts drawn from recent news and records, sourced as LiveBench does. The second uses a knowledge-edited counterfactual in the style of MRKE, stated explicitly in context so the correct answer is defined. Twins keep the seed's relation type, number of hops and answer format. Entity popularity is matched through a frequency proxy computed on an open corpus.
Annotators work in pairs in a CoTEVer-style interface. The interface shows the question, the model's step-by-step chain and retrieved evidence side by side, following the parallel-alignment designs of Towards an Ideal Tool for Learner Error Annotation. For each chain, annotators mark the first erroneous step. They also classify it as a factual error, a logically invalid inference from correct premises, or an error carried forward from an earlier step. A third annotator adjudicates disagreements. Following PRM800K, we route annotator time preferentially to chains whose final answer is wrong but which current detectors score as confident. Those are exactly the cases the claim is about. We target about 3,000 twin pairs and step labels for roughly 12,000 chains across models. Inter-annotator agreement on the first-error position is reported as a first-class result.
The second component is a controlled contamination arm. It gives us contamination with known ground truth. Following the Fragility paper, we fine-tune Qwen2.5-7B-Instruct on CoT traces for a random half of the fresh twins, then optionally apply one epoch of clean GRPO. We do the same on DeepSeek-R1-Distill-Llama-8B using the Stage II recipe. The untouched half serves as the fresh control. Because both halves come from one pool, membership is the only difference between them.
The third component is the measurement. For each model, prompt style (direct, zero-shot CoT, least-to-most) and item condition (public seed, fresh twin, injected-contaminated twin), we run the detector families from the motivating study and compute AUROC and expected calibration error. The key statistic is the difference-in-differences between the CoT-induced AUROC drop on exposed items and the same drop on fresh items. We estimate it with item-level bootstrap intervals. For localization, we fit a lightweight probe on hidden states and token entropy. We apply it both at the final answer and at the human-marked first-error step, and compare the two. The probe is trained on some models and item conditions and tested on held-out ones.
Baselines
The naive reference point is the same detectors applied to direct answers with no CoT, together with plain answer perplexity. The motivating study reports that advanced detectors often lost their edge over perplexity under CoT. Beating perplexity is therefore the minimum bar for any detector we propose.
The strongest existing competitor is the consistency-based family: sample several chains and score how much their final answers agree. Chain-of-Thought Prompting Obscures Hallucination Cues identifies it as the most robust to CoT. It reports this only as counts of degraded configurations. We found no reported AUROC on Lacuna for consistency detectors under CoT on these benchmarks, so we re-run it ourselves. It gets the same models, items and sampling budget as our probe, meaning the same number of generated tokens per item. Our step-localized detector counts as beating it if it achieves higher AUROC on fresh twins under CoT with non-overlapping bootstrap intervals, at equal or lower generation cost.
The second competitor is automated step judging. An LLM labels each step against retrieved evidence, as HalluMeasure does for atomic claims. HalluMeasure reports a claim-level F1 rising from 0.76 to 0.85 with CoT, but only 0.33 macro-F1 on error subtypes. We run the same judge over our chains and use its first-error labels in place of human labels.
The third competitor is an off-the-shelf process reward model, a scorer trained to rate individual reasoning steps. We use a PRM800K-trained model and the adversarially trained PRM of Adversarial Training for Process Reward Models, which reports +3.4 points over ReST-MCTS on math. These models were trained on mathematics, so we expect weak transfer to factual steps, but they are the obvious ready-made step scorers.
Two ablations isolate our key ingredients. The first swaps human first-error labels for LLM-judge labels while holding the probe fixed. The second reads the probe at the final answer instead of the first-error step. The fresh-versus-contaminated contrast is itself the ablation for the contamination claim.
Why not just…?
Why not just use DyVal or BeyondBench, which are already contamination-free? They generate formal problems with solver-checkable steps. The CoT blind spot was reported on factual and commonsense QA, and the Snoopy result suggests that exposure effects are driven by facts and term frequency. We do include DyVal arithmetic and reachability tasks as a sanity arm. There, first-error steps can be computed automatically and checked against our human labels. The main question, however, requires factual twins, and no solver produces those.
Why not just let an LLM judge mark the first wrong step and skip the humans? This is a real threat. If the judge agrees with adjudicated human labels at a rate close to human–human agreement, the collaborative annotation layer is unnecessary, and we will report that result and drop it. Two pieces of evidence suggest it will fall short, especially on fresh twins. LiveBench shows judges failing on items they cannot solve. HalluMeasure's 0.33 subtype macro-F1 shows that judges struggle to tell kinds of error apart, and separating factual from logical errors is exactly what the localization claim needs. The label-swap ablation settles the question directly.
Why not just use consistency-based detectors, since they already resist CoT? They may remain the best option. The contribution then becomes the measured contamination confound, not a new detector. The experiment reports how their robustness changes between public and fresh items. If they are equally robust on both, our interaction claim fails for that family, which is informative in its own right.
Why not just split public benchmarks with a membership detector instead of building twins? The Fragility paper shows these detectors fall to 50–58% AUROC on CoT-contaminated reasoning models. Any split they produced would be close to random. Constructing twins and injecting contamination deliberately are the only ways to know which items the model has seen.
Decisive Checks
The contamination claim holds if the CoT-induced AUROC drop is significantly larger on injected-contaminated twins than on their fresh halves, and in the same direction for public seeds versus fresh twins. The claim must hold for the internal-state and self-evaluation families, the ones the motivating study found most damaged. It also has to survive popularity matching, meaning it persists within popularity bins. The claim is refuted if the drop is the same size on fresh items. In that case the blind spot is intrinsic to CoT, and the step-localized detector carries the paper.
The localization claim holds if probe AUROC at the first-error step beats both the final-answer probe and repeated-sampling consistency on fresh CoT items. It also requires that this advantage shrinks on contaminated items, where memorized chains should look confident at every step. We will additionally check whether the GRPO stage erases the contamination effect on hallucination detection, as it erases the effect on membership detection. If it does not, hallucination-detector distortion becomes a candidate signal of contamination that survives RL cloaking, a use no current membership detector offers.
References
- Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
- CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
- Let's Verify Step by Step
- Towards an Ideal Tool for Learner Error Annotation
- HalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning
- Adversarial Training for Process Reward Models