Borrowed Confidence: Catching Distractor-Induced Reasoning Errors by Contrasting Contexts

Many systems now let a language model check its own work while it writes. They watch for moments when the model becomes unsure, meaning its prediction for the next word is spread across many options, and treat those moments as likely errors. This works on clean math problems. Real prompts are rarely clean, though. They contain retrieved documents, backstory, and stray numbers. Evidence from several papers suggests that this kind of irrelevant context makes models wrong in a confident way. They copy a number from an irrelevant sentence, or they build a fluent justification for a misleading claim, and their next-word predictions stay sharp the whole time. If that is right, the uncertainty alarm stays silent in exactly the setting where it is most needed.

We propose to first measure how badly uncertainty-based error detectors degrade as distractor density rises. We then test a cheap replacement signal that we call borrowed confidence. Each reasoning step is scored twice: once given the full noisy prompt, and once given a stripped-down restatement of the problem that the model writes itself. A step that is confident under the full prompt but surprising under the restatement is drawing its confidence from material the model itself judged irrelevant. That pattern is what a distractor-induced error should look like. In two sentences: entropy cannot see errors that the context makes look easy, but comparing how likely each step is with and without the noise can. We test whether that comparison detects and repairs distractor-induced errors better than entropy, weight-perturbation uncertainty, or simply throwing the noise away.

Core Claim

The claim has two parts, and the second matters only if the first holds. The first is diagnostic. On benchmarks where distractor density is controlled, the error-detection quality of token-level uncertainty (entropy, perplexity, and more careful epistemic estimates) drops as distractors increase. The drop should come specifically from errors whose source is a distractor sentence, because these are produced with low entropy. The second part is methodological. A per-step contrast between the full context and the model's own pruned context recovers most of that lost detection quality. It can also be used during decoding to roll back and regenerate the offending step, and it should beat both full-context and pruned-context generation alone.

Let $x$ be the full prompt, $\tilde{x}$ the model's own restatement keeping only the facts it judges necessary, and $u = (y_a, \dots, y_b)$ one reasoning unit (a sentence or equation) generated after prefix $y_{<a}$. The borrowed-confidence score of the unit is

$$ B(u) = \frac{1}{b-a+1}\sum_{t=a}^{b}\Big[\log p(y_t \mid y_{<t}, x) - \log p(y_t \mid y_{<t}, \tilde{x})\Big], $$

which is the average per-token log-probability advantage the full context gives the step. A unit is flagged when it is confident under $x$ (low entropy) but $B(u)$ is large. Computing $\tilde{x}$ takes one extra short generation. Scoring the already-generated tokens under $\tilde{x}$ is a single teacher-forced forward pass over a second cache, so the cost is about twice that of a forward pass. It needs no ensemble of weight variants.

Where This Sits

Three papers motivate the diagnostic. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering shows that zero-shot readers assign probabilities like 0.98 to answers lifted from irrelevant passages. Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning shows that step-by-step reasoning models defend misleading snippets with long rationalizations: Qwen3-8B falls from 96.22% clean accuracy to 77.03% with thinking on and 74.87% with thinking off. How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark shows that distractors erode step-level arithmetic, with Grok-3-Beta step accuracy falling from 43% with one distractor to 19% with fifteen. None of these papers asks whether internal uncertainty signals stay informative in these conditions.

The detectors we stress-test all assume that errors are uncertain and evaluate on clean prompts:

  • Once-More uses perplexity over reasoning units to trigger a verifier and logit redistribution. Its authors themselves name "confidently wrong" steps as the case where the trigger fails.
  • Reflection-Window Decoding pauses on entropy spikes within a short trailing window.
  • TokUR separates epistemic uncertainty (doubt from the model's own lack of knowledge) using random low-rank weight perturbations.
  • Estimating LLM Uncertainty with Logits reads evidence strength from raw logits in one pass.

These supply the baselines and, in Once-More's case, the repair mechanism we reuse. Our contribution is to replace the trigger, not the repair.

The closest method is LUMINA. It detects retrieval-augmented hallucinations by measuring how much the output distribution shifts when the context is swapped for random documents, combined with a layer-wise internal-knowledge signal. LUMINA classifies whole responses after generation, uses externally chosen random documents as the counterfactual, and targets short question answering. We contrast against the model's own minimal restatement, score each reasoning step during decoding, and use the score to choose where to roll back in multi-step reasoning.

Aligning Extraction and Generation for Robust Retrieval-Augmented Generation and Less is More: Summary of Long Instructions is Better for Program Synthesis motivate the restatement itself: having the model extract relevant content helps. They also constrain the design, because they make "just prune" a serious competitor that we must beat rather than wave away.

Research Program

The first stage builds error labels at the level of individual steps that say where each error came from. GSM-DC's dependency graphs specify which premises a correct solution needs. That lets us automatically sort each erroneous step into one of two kinds:

  • distractor-sourced: it uses a quantity or entity that appears only in an injected sentence;
  • intrinsic: an arithmetic or logic slip using only relevant premises.

On CogniLoad the generator likewise marks which statements are distractors, and we sweep its distractor-density parameter across the range where the paper reports a U-shaped accuracy curve. For SciAux (billhdzhao/SciAux) labels exist only at the answer level, split by helpful, irrelevant, and misleading snippets.

The second stage is the diagnostic. For each detector we compute step-level and answer-level detection quality, separately for distractor-sourced and intrinsic errors, as a function of distractor density. We also plot the entropy distribution of each error type. The prediction is that distractor-sourced errors have entropy indistinguishable from correct steps. If that is false, the premise is wrong and the program stops here with a useful negative result.

The third stage evaluates $B(u)$ as a detector, alone and combined with entropy through a two-feature logistic model fit on a small held-out split.

The fourth stage uses it as a trigger inside Once-More-style repair. When a unit is flagged, we regenerate it with logit redistribution against the flagged tokens, conditioning on $\tilde{x}$ for that unit only and then returning to $x$. This lets a fact that the pruning wrongly dropped re-enter later steps.

The main models are Qwen3-8B, which allows direct comparison to the SciAux numbers, and Llama-3.1-8B, with Llama-3.2-1B for comparison to TokUR's reported results.

Baselines

Plain decoding. The do-nothing reference is ordinary greedy decoding on the full noisy prompt, with no detection. Its accuracy defines the noise penalty that every method tries to close.

Token entropy and unit perplexity. These are the triggers used by Reflection-Window Decoding and Once-More, plus the minimum-token-probability rule that Language Model Cascades: Token-Level Uncertainty And Beyond argues beats averaging. They are free to compute and are the incumbents.

TokUR. TokUR is the strongest reported step-level detector in the file. It reaches 80.64% AUROC on MATH500 with Llama-3.2-1B, against 55.41% for log-likelihood and 71.17% for self-certainty. That number is on clean problems. We found no reported result on Lacuna for TokUR, or any token-level detector, on GSM-DC, CogniLoad, or SciAux, so we run it ourselves.

LogTokU. LogTokU is the single-pass epistemic baseline, reported at 71.78% AUROC on TruthfulQA with LLaMA-2-7B.

Running the detectors fairly. All detectors score the same generated trajectories, and thresholds are tuned on the same held-out split. TokUR's ensemble size is matched to roughly our two-pass cost and also run at its default size, so we can report both its matched-compute and best-effort results.

Repair baselines. For the repair stage, the strongest competitor is Once-More with its perplexity trigger. Its reported AIME 2024 gain for Qwen3-14B is 26.7% to 36.7%. On distractor benchmarks there is no reported result on Lacuna. Our only change is the trigger, so any difference is attributable to it.

Pruned-only generation. This baseline solves the problem conditioned only on $\tilde{x}$. It isolates whether the contrast adds anything beyond the restatement. We also run Ext2Gen where its released model allows. Its reported score under noisy retrieval with Llama3.1-8b-instruct is 0.557 against 0.473 for the default model, on its question-answering tasks rather than ours.

Ablation with random pruning. Here $\tilde{x}$ is replaced by a restatement with randomly deleted sentences of the same length. This tests whether the model's own relevance judgment is the ingredient doing the work, or whether any context perturbation would do.

What counts as beating the baselines. On distractor-sourced errors, $B(u)$ combined with entropy must exceed TokUR's step-level AUROC by a clear margin, with confidence intervals from bootstrap over problems. It must also not fall below entropy on intrinsic errors or on clean GSM8K and MATH500. In the repair stage, the borrowed-confidence trigger must exceed both full-context Once-More and pruned-only generation in accuracy, at comparable or lower token cost.

Why not just…?

Why not just prune the context and solve the clean version? This is the real threat. If the model's restatement were always complete, conditioning on it would make the contrast unnecessary. Ext2Gen's results suggest that pruning inside generation is strong. UncertaintyRAG and the multi-hop setting suggest pruning drops needed evidence. The pruned-only baseline settles this directly. We also report results stratified by whether $\tilde{x}$ retained all premises that GSM-DC's graph marks as required. The idea earns its place only if the hybrid wins on problems where pruning lost a premise and ties where pruning was perfect. If pruned-only matches the hybrid overall, we will report that pruning is sufficient, and the detection result alone stands as the contribution.

Why not just recalibrate with temperature scaling? Improving Semantic Uncertainty Quantification in Question Answering via Token-Level Temperature Scaling shows that one global temperature is a strong calibrator, raising TriviaQA AUROC from 0.761 to 0.865. But one temperature applied to every token cannot reorder which steps look confident relative to one another. If distractor-sourced errors sit at the same low entropy as correct steps, rescaling leaves them indistinguishable. We include temperature-scaled entropy, tuned on clean and on noisy validation data, as a detector to show this rather than assert it.

Why not just use self-consistency voting? Distractors pull models toward the same wrong path, as GSM-DC's systematic error patterns suggest, so voting may amplify the error instead of averaging it out. We run self-consistency at a sample count matched to our token budget and at 16 samples. We report the fraction of problems where the majority answer is distractor-sourced; that fraction is what tests the systematic-pull explanation.

Why not just ask the model to falsify its answer? Agentic Uncertainty Reveals Agentic Overconfidence cut Claude's calibration error from 0.37 to 0.24 by asking it to look for bugs rather than confirm correctness. Thinking in a Crowd, however, shows that extra deliberation under misleading context deepens rationalization. We run a "find the irrelevant fact you used" post-hoc check as a baseline at matched tokens. It works without access to logits, which is a practical advantage we acknowledge.

Decisive Checks

The program hinges on three results.

The first is the error-type split of entropy on GSM-DC and CogniLoad. If distractor-sourced errors are not low-entropy, the premise fails.

The second is the detection comparison on distractor-sourced errors across density levels. The specific prediction is that the gap between $B(u)$ and entropy-based detectors widens as density grows. It should track CogniLoad's reported region of worst accuracy at intermediate densities.

The third is the repair comparison against pruned-only generation, stratified by restatement completeness, on GSM-DC, GSM-Plus Distract, and SciAux. We report answer accuracy, step accuracy, tokens generated, and expected calibration error of the final answer confidence. We also check that the clean-input baselines (GSM8K, MATH500) do not lose accuracy.

We expect SciAux's misleading snippets to be the hardest case. A misleading sentence looks relevant, so the model may keep it in $\tilde{x}$, and the contrast would then be blind. A small gain there, alongside large gains on irrelevant distractors, would be the honest and informative outcome. It would mark the boundary between noise the model can recognize as irrelevant and noise it cannot.