Reasoning models are easily derailed by irrelevant or misleading sentences in their prompt. When a math problem mentions that five kiwis were smaller than average, a model will often subtract five anyway. Step-by-step graders, which score each intermediate step instead of only the final answer, do not catch this. They judge whether a step follows. They do not judge whether the facts it uses were licensed by the question.
We propose to generate step labels with a structured argument between language-model agents, where the argument is about one thing: provenance. A skeptic challenges each quantity or claim a step uses. A defender must quote the exact sentence of the problem, or an earlier accepted step, that justifies it. A judge sees only the problem, the quote, and the step, never the full persuasive reasoning trace, and rules on whether that justification holds. These labels are distilled into a cheap step grader and used as a dense reward during reinforcement learning. The goal is models that keep their long-form "thinking" but stop using it to rationalize noise.
The opportunity matters because three findings in the literature fit together. Long reasoning makes contextual noise worse, not better. Existing step graders barely look at the prompt. Human step annotation does not scale to the many kinds of noise a deployed model meets. A labeler that checks grounding, and costs no human time, attacks all three at once.
Core Claim
Step-level supervision fails under contextual noise because its labels conflate two properties. The first is local validity: the arithmetic or inference is correct. The second is grounding: every premise the step uses traces to a relevant part of the problem. Current graders measure mostly the first. Reward Under Attack: Evaluating the Sensitivity of Process Reward Models audits open process reward models (PRMs, models that score each reasoning step) under text perturbations. It finds that Qwen2.5-Math-PRM-7B's reward barely moves when the question is deleted entirely (shift peaking near 0), and that fluent steps containing errors are often rewarded. A step that correctly subtracts an irrelevant number is locally valid and ungrounded, and such a grader cannot tell.
We claim that an information-asymmetric provenance debate produces a separate, reliable grounding label without human annotation. It should do better than a single critic, because the critic that rules never reads the rationalization. We further claim that rewarding grounded steps during reinforcement learning reduces distraction on noise types never seen in training, without the clean-accuracy cost of turning reasoning off.
Why This Is Not Already Done
The closest work splits into three camps, and each leaves the grounding question untouched.
ChainLM has a committee of personas debate each reasoning step at inference time, with a judge gating progress. It raises Elementary Mathematics accuracy from 33.33% to 44.44%. Its debate asks "is this step correct?" rather than "where did this premise come from?" It runs only at inference, multiplying latency, and it never tests prompts containing distractors.
Generative Adversarial Reasoner (GAR) and Adversarial Training for Process Reward Models (APRM) remove the human-annotation bottleneck by training a reasoner, or error generator, against a single learned critic. GAR's critic reads ~320-token slices and averages binary verdicts into one trajectory reward. Both are two-player setups whose critic reads the full trace. We reuse GAR's slicing and reinforcement-learning recipe, but change who judges and what the judge sees.
Multi-Agent Debate with Memory Masking shows that debate collapses when earlier turns contain confident errors. It fixes this by masking whole responses. We borrow the masking idea at step granularity: a step rejected on provenance grounds becomes uncitable for later steps, so the defender cannot build on it.
Two papers motivate the focus on noise. Thinking in a Crowd documents a "rationalization trap": a model given misleading auxiliary snippets spends its reasoning tokens building bridges from the false premise to a wrong answer. Distractor Injection Attacks on Large Reasoning Models shows that models trained with reinforcement learning on verifiable answers are more vulnerable to injected side-quests. Both findings imply that any critic reading the full persuasive trace is exposed to the same trap, which is why the judge's isolation is the central design choice here.
Research Program
Segmentation and claim extraction. Each sampled reasoning trace is segmented into steps, using GAR-style slices for long traces. A small extraction prompt lists, for each step, the quantities and factual claims it consumes.
The debate. For each consumed item, the skeptic sees the problem and the step and issues a challenge. The defender sees the full trace and the problem and must return a verbatim quote, either from the problem or from a previously accepted step. The quote's existence is checked mechanically by string match, so fabricated citations fail at no model cost.
The judge receives only three things: the problem, the quoted span, and the step. It answers two questions. Does the span support this use? Is the span relevant to what the question asks? A separate local-validity check rules on the step's inference given its premises. Each step thus receives a pair of verdicts, one for validity and one for grounding.
Masking. Steps that fail on grounding are removed from the set of citable prior steps. This is step-level masking in the spirit of MAD-M², so contamination cannot be laundered through later steps.
Distillation and training. Because debate is expensive, it runs offline on a fixed labeling budget. We train two systems on its labels:
- A 7B step grader that predicts both verdicts.
- A policy trained with Group Relative Policy Optimization (GRPO, the reinforcement-learning algorithm GAR uses) on outcome reward plus a dense step term.
To address the credit-assignment weakness of GAR's averaging, the dense term rewards the trajectory by the fraction of steps before the first ungrounded or invalid step, rather than by an average. The grader is periodically re-labeled by fresh debates on current policy samples, because static graders degrade as the policy's style shifts, as DC-W2S reports.
Data and backbones. Training problems come from OpenR1-Math-220k, with distractors injected at varied positions (start, middle, end), since position matters per the Distractor Injection study. Backbones are DeepSeek-R1-Distill-Qwen-7B, matching GAR, and Qwen3-8B, matching the Distractor Injection and SciAux studies.
Validating the labels. We compare against a small human-audited set of roughly 500 steps drawn from semantic-distractor and misleading-premise problems. This is the only human labeling in the program. It is used to evaluate the labels, not to train on them.
Baselines
Outcome-only reinforcement learning. The do-nothing reference is the same backbone trained with GRPO on outcome reward alone, using identical problems (including injected distractors), identical steps, and identical rollout counts. This isolates whether any step signal helps. Beating it means recovering at least half of the gap between clean and noisy accuracy on GSM-NoOp and SciAux misleading-context items, while staying within one point on clean MATH-500 and AIME24.
GAR with a single critic. The strongest existing method for annotation-free step reward is GAR. It reports AIME24 rising from 54.0% to 61.3% for DeepSeek-R1-Distill-Qwen-7B. We rerun GAR's single-critic slice reward on our distractor-injected training mix, giving its critic the same total token budget as our three debate roles. We spend the extra budget on several sampled critic verdicts per slice combined by vote. Beating it means higher noisy-context accuracy with no AIME24 regression, and higher agreement with the human-audited grounding labels.
SFT followed by Direct Preference Optimization (DPO, fine-tuning on preferred-versus-rejected trace pairs). This is the strongest noise-specific defense. Distractor Injection Attacks reports it restoring Qwen-3-8B's AIME score under attack from 4.9% to 57.8%. We rebuild it on our injected training set with matched compute. Because it is trained on the same injection family, the fair test of our method is transfer to noise it did not see. That transfer suite comprises SciAux's natural-language misleading snippets, the GSM-Plus critical-thinking (missing-information) split, and GSM-Symbolic's No-Op clauses if these were excluded from training. Matching it in-distribution and beating it on transfer counts as a win.
AbstRaL. AbstRaL strips flavor text by rewriting problems symbolically and solving with SymPy. It reports Qwen2.5-0.5B rising from 34.0% to 44.6% on GSM-Symbolic. We report it on GSM-Symbolic and GSM-Plus as an upper reference for symbolic domains, not as a like-for-like competitor on SciAux, where no symbolic form exists.
Ablations isolating the key ingredient. The first ablation keeps all three roles but shows the judge the full trace. If gains vanish, the isolation is doing the work. The second collapses validity and grounding into one label. The third disables step-level masking.
Why not just…?
Why not just label steps by string-matching the injected distractor numbers? For synthetic numeric No-Op clauses this is nearly free and probably as accurate as debate, and we will use it as a check on the debate labels for exactly that case. It fails wherever the distractor is not a literal token. Examples are a misleading scientific claim in SciAux, a narrative detail misread as a requirement, or a premise that is reworded before being used. Our transfer evaluation is built around these cases. If string-match-labeled training matches our method on SciAux and GSM-Plus critical-thinking, the debate is unnecessary and we will say so.
Why not just ask one critic to check provenance? This is the most serious threat, and the full-trace ablation together with the token-matched GAR baseline is designed to settle it. The argument for separating roles comes from Thinking in a Crowd: models reading a persuasive chain that incorporates misleading context tend to justify it. The judge's isolation from that chain is the mechanism we are testing.
Can LLM Agents Really Debate? cautions that debate is capped by the strongest agent's reasoning, and that agents in an incorrect majority stay put over 97% of the time. Our design avoids majority dynamics: the judge rules on quoted evidence, not on votes. We still expect the cap to apply, so we report results by backbone strength.
Why not just turn thinking off when context looks noisy? Thinking mode adds roughly 10–13% accuracy on SciAux without noise. Turning it off throws away that gain on every input flagged as noisy, and it requires a reliable noise detector that does not exist. The file's summary of the SciAux result is also internally inconsistent. It reports Qwen3-8B at 77.03% with thinking on and 74.87% with thinking off under two misleading snippets, while describing thinking-on as worse. We will re-measure both modes ourselves and include thinking-off as a baseline under matched conditions, rather than rely on the summary.
Why not just majority-vote over sampled answers? A Diagnostic Study of Multi-Agent LLMs for Real-World Debates finds that when the initial majority is wrong, low-interaction setups recover the correct answer about 40% of the time, versus 74% for high-process deliberation. Distractors bias all samples in the same direction, which is exactly the regime where voting fails. We include self-consistency at matched inference tokens as a reference point.
Proposed Experiments and Decisive Checks
Benchmarks. Evaluation covers four groups:
- GSM-Symbolic and its No-Op variant. Phi-3-mini is reported there with a 65% drop and o1-preview with a 17.5% decline.
- GSM-Plus distractor-insertion and critical-thinking splits. GPT-4 shows an 8.23% performance drop rate and open models 30–40%.
- SciAux helpful, irrelevant, and misleading conditions.
- AIME24 and MATH-500, both clean and with injected distractors at three positions.
For the noise benchmarks, the main metric is the drop from clean to noisy accuracy. Clean accuracy guards against regression.
Grader diagnostics. We probe the distilled grader with the PRMProbe-style perturbations from Reward Under Attack and with ProcessBench.
The first decisive check concerns grounding. If our grader truly reads the problem, deleting or shuffling the question must sharply lower its step rewards. This is where Qwen2.5-Math-PRM-7B's shift peaks near 0. At the same time, semantics-preserving rewrites must leave rewards nearly unchanged; the published reference is a shift under 0.02.
The second decisive check is the full-trace ablation. If showing the judge the whole trace lowers agreement with the human-audited labels on misleading-premise steps, the information-asymmetry hypothesis is supported. If not, the method reduces to a provenance-prompted single critic, and we report that.
The third decisive check is transfer. Training only on injected math, our policy should shrink the misleading-context gap on SciAux more than SFT+DPO and string-match labeling do. If it does not, grounding supervision is learning the injection template rather than a habit of checking premises.
References
- Reward Under Attack: Evaluating the Sensitivity of Process Reward Models
- ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
- Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
- Adversarial Training for Process Reward Models
- Multi-Agent Debate with Memory Masking
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense
- DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoning
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
- Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
- A Diagnostic Study of Multi-Agent LLMs for Real-World Debates