Reasoning models fail in a specific way when a prompt contains misleading or irrelevant text. They do not skip over the bad passage. They spend their step-by-step reasoning building a justification for it. Diagnostic studies show that turning on "thinking" can make a model worse than answering directly once two misleading snippets are present. Reinforcement learning with verifiable rewards trains models with a programmatic pass/fail check on the final answer, and it is the natural tool for fixing this. The trouble is that a pass/fail signal is a poor teacher here. When a model fails every sample on a distracted prompt, the standard group-relative update has nothing to push against. When it passes, the reward cannot tell "ignored the distractor" apart from "ignored the context entirely."
We propose to keep the reward binary and change what each rollout is compared against. For each question we sample responses on several context variants at once: the clean question, versions buried in distractors, and versions where the context legitimately changes the correct answer. We then compute the learning signal relative to the model's success across all of these siblings. A failure under distraction is penalized in proportion to how often the model solves the clean version, so the update targets distraction-induced failure rather than plain inability. The variants whose correct answer depends on the context make the cheap fix, ignoring all context, earn reward zero. The method needs no learned reward model and no step-level grader.
Core Claim
Outcome-only training on distracted prompts has two failure modes that simple data augmentation does not address.
The first is gradient starvation. Standard group-relative policy optimization (GRPO) normalizes each rollout's 0/1 reward by the mean and standard deviation of rewards within its prompt's group. Distracted prompts that the model consistently fails therefore contribute zero gradient. This is exactly the variance collapse described in Discounted Beta–Bernoulli Reward Estimation, and distractors produce it on purpose.
The second is shortcut learning. A model can raise its pass rate on distracted prompts by discounting context wholesale, leaning on what it already knows. Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning shows that outcome-only rewards readily produce reasoning that does not cause the answer, and nothing in a distractor-augmented reward prevents the same thing from happening with context.
Our claim is that pooling the group-relative comparison across context siblings of one question addresses both failure modes with the same binary verifier, and that this cannot be achieved by adding distractors to the training data or by smoothing rewards over time.
Mechanism
Let $q$ be a question and $c_1,\dots,c_K$ its context variants. Each variant $c_k$ has its own verifiable answer $y_k$; for distractor variants $y_k$ equals the clean answer, and for "evidence-dependent" variants it is recomputed from the altered context. We draw $m$ rollouts per variant, so $G = Km$ rollouts per question, the same budget as standard GRPO with group size $G$. Each rollout $\tau_{k,i}$ receives $r_{k,i} = \mathbb{1}[\text{answer}(\tau_{k,i}) = y_k]$. The pooled advantage is
$$\hat{A}_{k,i} = \frac{r_{k,i} - \bar{r}_q}{\sigma_q + \epsilon},$$
where $\bar{r}_q$ and $\sigma_q$ are the mean and standard deviation of all $Km$ rewards for question $q$, and $\epsilon$ is a small constant.
Two consequences follow directly from the formula. First, if the model solves the clean variant but fails every distracted variant, $\sigma_q > 0$, so the distracted failures receive a negative advantage. Its size grows with clean-variant success, which is the counterfactual baseline we want. If the model cannot solve the question even when clean, $\bar{r}_q$ is low and distracted failures are barely penalized, so the update does not chase unsolvable items. Second, a policy that ignores context passes the clean and distractor variants but fails the evidence-dependent ones. It therefore never reaches the all-ones group that standard GRPO would treat as uninformative, and it keeps getting pushed toward reading the context selectively.
The ratio of variant types is the main design choice. We set it so that "always ignore context" and "always trust context" each yield an expected group reward well below 1.
Research Program
Training data. The training environment is built from tasks whose answers can be recomputed by a program when a premise changes. ReasoningGym tasks are procedurally generated, so we move one operative parameter of each instance into a short "document." We then embed that document among irrelevant documents and misleading documents that state different values for similar-looking entities. For evidence-dependent variants we alter the parameter and regenerate the answer. For science question answering we use the helpful, irrelevant, and misleading auxiliary conditions of SciAux (; [art\_e5ba7d32c8e7450596d2e9442ee21a24](/paper/art-e5ba7d32c8e7450596d2e9442ee21a24/art_e5ba7d32c8e7450596d2e9442ee21a24)), introduced by Thinking in a Crowd. We hold out a disjoint question split for evaluation.
Distractor generation. Distractors are generated in two strengths. Static distractors are sampled documents. Adaptive distractors come from a tree search in the style of Breaking Focus, run against a frozen copy of the starting policy. We do not run a co-evolving adversary in the main study. Gap 2 in the literature suggests that direction, but it would confound the effect of the grouping change with the effect of an escalating curriculum. We reserve it for a follow-up.
Models and diagnostics. The policies are Qwen3-1.7B, 4B, and 8B with thinking enabled. Beyond accuracy, we track three diagnostics on held-out data. The first is the Causal Importance of Reasoning (CIR) and Sufficiency of Reasoning (SR) measures from Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning, computed on distracted prompts, which tell us whether the reasoning is doing the work. The second is thinking-trace length under misleading context, the signature of the rationalization trap. The third is a context-sensitivity score: accuracy on evidence-dependent variants, which falls if the model learns to ignore context.
Role of prior work. The rationalization trap and the 45% average collapse under adaptive distraction motivate the problem. The two sources are Thinking in a Crowd, which is purely diagnostic and trains nothing, and Breaking Focus, which mitigates with offline preference pairs and short answers rather than online reasoning.
Lessons from Training Grounded LLMs with Verifiable Rewards is the closest prior work and also constrains our design. It applies GRPO with answer, citation-entailment, and refusal rewards over naturally noisy retrieval. It reports that the model's reasoning often recognizes that context is useless yet answers anyway. It also reports that harsh penalties make small models collapse into excessive caution. We read the second result as a warning that any distraction penalty must be counterbalanced by a reason to keep using context, which is the job of our evidence-dependent variants. Unlike that work, we change the comparison set rather than adding reward terms, and we use deliberately constructed distractors.
Rate or Fate? further constrains the design. It shows that GRPO under a noisy binary verifier learns only when true-positive rate minus false-positive rate is positive. Multiple-choice lucky guesses on distracted prompts act like false positives, so we report how often correct answers on distracted variants coincide with low-CIR reasoning, as a rough estimate of this effective noise.
Baselines
Untrained model. The naive reference is the starting model with thinking on and off. On SciAux, Thinking in a Crowd reports that Qwen3-8B scores 96.22% at baseline. With two misleading snippets it drops to 77.03% with thinking on and 74.87% with thinking off. Any trained method must close most of the gap to 96.22% without hurting accuracy on helpful-context variants.
Distractor-augmented GRPO. This is the strongest competitor. It is ordinary GRPO with group size $G$, trained on the same mixture of clean, distracted, and evidence-dependent prompts, but with each variant forming its own group. It sees identical data and an identical rollout budget, uses the same learning rate sweep, and runs for the same number of steps. It is the method a practitioner would try first. The comparison isolates exactly one thing: whether the advantage is computed over a single prompt or across sibling variants of the same question.
Augmented GRPO with temporal smoothing. This is augmented GRPO with the reward estimate replaced by the discounted Beta–Bernoulli estimate of Discounted Beta–Bernoulli Reward Estimation. That method fixes variance collapse by pooling a prompt's history over time rather than across contexts. On DAPO-Math-17k it reports gains of +3.43 (1.7B) and +2.32 (8B) points in in-distribution Acc@8 over GRPO. If it matches our method, the benefit was merely fixing zero-variance groups, not the counterfactual baseline.
Offline preference training. This is direct preference optimization (DPO), which trains directly on pairs of preferred and rejected responses, applied to distracted-versus-clean pairs as in Breaking Focus. The pairs are built from the starting policy's own rollouts on the same training questions, and we match total generated tokens. That paper reports that DPO substantially restores robustness but gives no single reusable figure for our setting. Its clean-to-adaptive CommonsenseQA collapse for GPT-4o-mini, from 85.7% to 22.0%, is the scale of attack we reproduce for evaluation.
Extract-then-generate. This is the Ext2Gen recipe from Aligning Extraction and Generation, which trains the model by DPO to quote evidence sentences before answering. It is used on our retrieval-style variants. On its own dataset it reports an average score of 0.557 against 0.473 for default Llama3.1-8B-Instruct.
Ablations. Two ablations isolate the two ingredients. The first keeps pooled advantages but drops the evidence-dependent variants. We predict it improves distracted accuracy but lowers context sensitivity. The second keeps evidence-dependent variants but uses per-variant groups. We predict it preserves context sensitivity but leaves distracted failures without gradient.
What counts as winning. Beating distractor-augmented GRPO requires both of the following on held-out questions at matched compute, over three seeds. First, higher accuracy on adaptive-distractor variants. Second, no loss on evidence-dependent and helpful-context variants.
Why not just…?
Why not just tell the model to ignore irrelevant text? Breaking Focus found that such instructions give negligible protection against coherent distractors. Thinking in a Crowd shows deliberation itself amplifies the problem. We include the instruction as a prompt variant of the untrained baseline, but we do not expect it to matter.
Why not just add distractors to the GRPO training set? This is the real threat, and the main comparison is designed to settle it. Our argument rests on two predictions. First, augmented GRPO wastes its updates on all-fail distracted groups. We will log the fraction of zero-variance groups for both methods. Second, whatever robustness it gains comes partly from discounting context, which would appear as a drop on evidence-dependent variants. If augmented GRPO matches us on both distracted accuracy and context sensitivity, the pooled baseline is unnecessary and we will report that.
Why not just fix variance collapse with temporal smoothing? The discounted Beta–Bernoulli estimate does give distracted failures a nonzero signal. However, its baseline is the same prompt's past success, not the same question's clean success. It therefore penalizes a hard distracted item and an item the model could solve cleanly in the same way. The smoothing baseline above tests whether that distinction matters. If it does not, the cheaper method wins.
Why not just filter the context upstream? Separate filters such as PRCA, a 0.4B adapter trained with ROUGE-L reward in front of a frozen reader, fail irrecoverably when they drop a needed premise. Ext2Gen already beat several such compressors. Adaptive distractors are also designed to look relevant to any filter. We therefore compare against the stronger in-model extraction baseline rather than an external filter.
Proposed Experiments
The first decisive check is the held-out SciAux misleading-context condition with thinking on. The question is whether pooled training lifts Qwen3-8B from the reported 77.03% toward its 96.22% clean accuracy, while the thinking-on minus thinking-off gap becomes positive again. That gap is the direct test of whether the rationalization trap has been removed rather than sidestepped.
The second check is transfer to attacks the model never trained against. We regenerate adaptive distractors with the tree search against each trained policy on CommonsenseQA, MMLU, and OpenbookQA. We then report the accuracy drop relative to clean accuracy, against the roughly 45% average drop that Breaking Focus reports for untrained models.
The third check is context sensitivity on the evidence-dependent ReasoningGym variants. Here the prediction separating our method from augmented GRPO is that augmented GRPO falls below the untrained model while pooled training does not.
The fourth check concerns reasoning quality. Using CIR and SR on distracted prompts, we test whether pooled training preserves or raises causal importance of reasoning, where Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning found outcome-only training lowers it on many tasks. We also test whether reasoning length under misleading context shrinks toward its clean-context level.
The method is falsified under either of two outcomes. One is that the temporal-smoothing baseline matches it on all four checks. The other is that the pooled-only ablation matches it on context sensitivity. In the first case the gain was merely from fixing zero-variance groups. In the second, the evidence-dependent variants are unnecessary.
References
- Discounted Beta–Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
- Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
- art_e5ba7d32c8e7450596d2e9442ee21a24
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Lessons from Training Grounded LLMs with Verifiable Rewards
- Rate or Fate? RLV$^{\varepsilon}$R: Reinforcement Learning with Verifiable Noisy Rewards
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter