Premise Audits: Teaching Models to Self-Correct When the Context Misleads Them

Language models answer worse when a prompt contains plausible but irrelevant or misleading text. Asking them to "check their work" does not help, because the second attempt rereads the same misleading text and usually lands on the same wrong answer. We propose to train a different correction move. On its second turn, the model names the passage in the context that its first answer relied on, decides whether the question actually supports that passage, and only then solves the problem again. The training signal for this move comes from a light human review loop. A model proposes which passage caused each of its own real failures, and a person accepts or corrects that label. Our claim is that supervision at the level of the blamed passage, gathered on the model's own mistakes, teaches self-correction that carries over to new kinds of distraction. Outcome-only training and synthetic error injection do not.

This matters because two recent literatures predict the same failure without having tested it together. Distraction studies show accuracy collapsing under coherent noise. Self-correction studies show that models rarely detect their own errors. Yet the one method that makes intrinsic self-correction work, reinforcement learning over two attempts, was evaluated only on clean inputs. If correction breaks precisely when the context is the cause of the error, current self-correction results overstate real-world reliability. A method that fixes this would be useful wherever retrieved documents or tool outputs are noisy.

Core Claim

Under distracting context, intrinsic self-correction fails for a specific and fixable reason. The correction turn is conditioned on the same distractor that caused the first error, so it re-derives the error. The distractor's content is what matters here; the model's reasoning skill is not the bottleneck. Our first testable claim is diagnostic. On noise-perturbed versions of standard benchmarks, the gain from the second attempt shrinks or turns negative, even for models trained to self-correct. The second claim is constructive. Adding a small amount of human-verified "which passage misled you" supervision on on-policy failures, on top of outcome-reward reinforcement learning, recovers positive correction gains. It also generalizes to distraction types never seen in training. That generalization is the test that Synthetic Error Injection Fails to Elicit Self-Correction In Language Models showed synthetic errors fail.

Where This Sits

Three papers motivate the gap.

  • Distraction breaks reasoning, and thinking longer makes it worse. Thinking in a Crowd built SciAux, which adds misleading and irrelevant snippets to ScienceQA. Step-by-step "thinking" modes do not filter misleading context. They produce longer traces that rationalize it: Qwen3-8B falls from 96.22% clean accuracy to 77.03% under two misleading snippets with thinking on.
  • Adaptive distractors are far stronger than static ones. Breaking Focus and Adaptive Distraction use tree search to generate coherent distractors tailored to each model. Accuracy drops by over 45% on average. GPT-4o-mini on CommonsenseQA falls from 85.7% to 22.0%.
  • Self-verification is unreliable. On the self-verification limitations of large language models on reasoning and planning tasks shows GPT-4 self-critique degrading accuracy, for example Graph Coloring from 16% to 2%.

None of these three studies correction under noise.

Two papers constrain the design.

  • Offline synthetic errors do not transfer. The error-injection paper fine-tuned models on synthetically corrupted reasoning. Models corrected up to 99% of synthetic errors but recognized only 8% of their own Sudoku mistakes. They also often said "I made a mistake" and then repeated the same token, which the paper calls the Parrot Problem. We therefore insist on on-policy failures, meaning errors the model actually made.
  • Self-correction gains vanish under fair evaluation. When Can LLMs Actually Correct Their Own Mistakes? finds that intrinsic gains disappear when the first attempt is given a best-effort prompt. It also finds that models fix errors far better once an external source points to the error. The premise audit is designed to make the model supply that pointer to itself.

Three lines of work supply methods.

  • Two-turn reinforcement learning. Training Language Models to Self-Correct via Reinforcement Learning (SCoRe) trains two attempts, reporting +15.6% absolute on MATH for Gemini 1.5 Flash and +9.1% on HumanEval. We adopt its two-stage training as our backbone and add one ingredient to it.
  • Author–reviewer annotation. Work such as Building a Role Specified Open-Domain Dialogue System found reviewing model drafts about 13 times faster than writing from scratch. This justifies having the model propose blame labels and humans only verify them.
  • Mechanistic measurement. Llama See, Llama Do finds that 3–10% of attention heads boost any token seen in the prompt, a behavior it calls entrainment. We reuse its differentiable-masking measurement to check mechanism.

The closest existing combination is the DPO step in Adaptive Distraction. DPO, or direct preference optimization, trains the model to prefer one response over another. Adaptive Distraction trains single-turn preferences between answers on distracted and clean inputs. It never asks the model to recover from its own distracted answer, and it gives no signal about which passage was at fault.

Research Program

Stage 1: diagnostic. We build noisy versions of correction benchmarks from three sources:

  • SciAux's misleading and irrelevant snippets;
  • tree-search distractors generated with the Adaptive Distraction procedure on MATH-500 and CommonsenseQA;
  • the same distractors attached to CorrectBench items (Can LLMs Correct Themselves?).

For base models (Qwen3-8B with thinking on and off, Llama-3.1-8B) and a SCoRe-style model trained on clean data, we measure four quantities. The first is the change in accuracy from attempt one to attempt two. The second is the rate at which a wrong answer becomes right. The third is the rate at which a right answer becomes wrong. The fourth is a "distractor-parrot" rate: how often the second answer still agrees with the answer the distractor implies.

Stage 2: data. We run Qwen3-8B on the noisy training splits and keep its genuinely wrong attempts. For each failure, a model proposes the context passage that the reasoning trace adopted. A human reviewer then does three things:

  • confirms that the distractor leaves the correct answer unchanged. This matters because tree-search distractors can silently alter the question, and invalid distractors would inflate measured damage.
  • accepts or edits the blamed passage.
  • marks whether the passage is misleading or merely irrelevant.

We cap the budget at roughly 2,000 reviewed failures and log reviewer minutes per item.

Stage 3: training. We run SCoRe's two-stage training on the noisy prompts. The second turn must open with a premise audit before the new answer. The reward on turn two is

$$r = \mathbb{1}[\text{final answer correct}] + \lambda \cdot \mathrm{F1}(\hat{s}, s^{*}),$$

  • $\hat{s}$ is the passage the model blames;
  • $s^{*}$ is the human-verified passage;
  • F1 is token overlap between the two;
  • $\lambda$ weights the audit term.

The audit term is applied only on the reviewed subset. The remaining prompts receive outcome reward alone.

Baselines

Naive reference points. The first is the base model answering once, with no second attempt. The second is the base model with a generic "review your answer and fix any mistakes" prompt. These anchor whether any correction happens at all. The second one is expected to have a near-zero or negative gain, as the self-correction survey and the GPT-4 verification study both found on clean data.

Defensive prompting. This is the base model told to ignore irrelevant information and answer "unanswerable" when unsure, a variant of the instruction used in Improving Zero-shot Reader by Reducing Distractions. That work reports a 64% average Exact Match gain over naive readers in open-domain QA. Adaptive Distraction reports that "ignore irrelevant information" instructions give almost no benefit against tree-search distractors. We run it with an identical decoding budget.

Self-consistency. This samples several answers and takes a majority vote, matched in total generated tokens to our two-turn method. The self-correction survey argues it beats iterative correction on many tasks. We found no reported result on Lacuna for self-consistency under adaptive distraction.

DPO on distracted versus clean pairs. This is the Adaptive Distraction recipe, which reports accuracy on perturbed inputs improving by nearly 50% in some smaller models. We train it on the same noisy prompts and the same human-validated subset, so that distractor-validity filtering is shared across methods.

Outcome-only SCoRe on noisy prompts. This is the strongest competitor, because it already has on-policy two-turn training. SCoRe's reported numbers are clean-data only: +15.6% absolute on MATH and +9.1% on HumanEval. We found no reported result on Lacuna for SCoRe under contextual noise. It receives identical prompts, reinforcement learning steps, seeds, and hyperparameter search budget. The only difference from our method is the audit term.

Two ablations isolate the key ingredient. In the first, audit spans are proposed by the model and never human-reviewed. In the second, audit spans are known by construction: SciAux records which snippet was inserted as misleading.

What counts as winning. On distraction types held out from training, we require the full method to beat outcome-only noisy SCoRe on turn-two accuracy and on wrong-to-right rate, beyond seed variance. It must do so without losing clean MATH accuracy.

Why not just…?

Why not just run SCoRe on noisy inputs, with no audits and no humans? This is the real threat, and it is the main comparison. Outcome reward on noisy prompts is already on-policy. It may learn to discount distractors implicitly. If it matches our method on held-out distraction types, the premise audit adds nothing and we will report that.

We expect it to fall short for two reasons:

  • the rationalization trap in Thinking in a Crowd suggests that longer reasoning over the same context entrenches the distractor rather than exposing it;
  • the self-correction survey finds models correct far better once told where the error is, which outcome reward never says.

Why not just DPO on distracted versus clean pairs, as Adaptive Distraction does? That recipe teaches which final answer to prefer, not how to recover from an answer the model has already committed to. Its reported gains are on the same perturbation family it trained on. Our held-out-distraction split tests whether the gains transfer. The shared validity filtering keeps the comparison about the training signal rather than data quality.

Why not just take a majority vote across samples? Self-consistency helps when errors are independent across samples. A coherent distractor biases every sample in the same direction. That shared bias is exactly what entrainment predicts. We test this directly with token-matched voting on the adaptive-distraction splits.

Why not skip humans, since SciAux already records which snippet is misleading? For SciAux this is largely correct, and the by-construction ablation will show it. Human review earns its cost on tree-search and CorrectBench distractors for two reasons:

  • no ground-truth blame exists there;
  • some distractors silently change the correct answer.

If model-proposed spans without review perform as well as reviewed ones, we drop the human role to validity checking only and say so.

Why not inject synthetic errors, which is cheaper? The error-injection paper already shows 99% synthetic correction collapsing to 8% recognition of natural errors, along with the Parrot Problem. We do not repeat that experiment.

Decisive Checks

The first check decides whether the problem is real. If outcome-only SCoRe trained on clean data still shows positive correction gains on noisy MATH-500 and SciAux, the premise of this proposal fails. The second check decides whether the audit is the remedy: premise audits must beat outcome-only noisy SCoRe on distraction types excluded from training.

The third check is mechanistic. Using differentiable masking as in Llama See, Llama Do, we test whether the audit step lowers the second-turn logit boost for tokens drawn from the blamed passage. If accuracy improves without this reduction, the model may be learning a shortcut rather than discounting the passage. The fourth check is cost-effectiveness. Reviewer minutes per item, compared against the gap between human-reviewed and model-proposed spans, tells us whether human review is worth paying for.