Language models trained with reinforcement learning on math and reasoning problems are easily thrown off by irrelevant sentences in the prompt. They copy a number from a side story, or they build a long justification around a misleading premise. The training method most used today, Group Relative Policy Optimization (GRPO), makes this hard to fix. GRPO samples several answers per problem and rewards each answer relative to the others. When every sampled answer to a distracted prompt is wrong, it computes no learning signal at all. Those are exactly the prompts where the model most needs correcting, and the expensive token-by-token generation that produced the answers is thrown away.
We propose recovering that wasted signal cheaply, and aiming it at the cause of the error. During training we build noisy prompts by inserting known irrelevant sentences into clean problems. For any failed answer, we can then rescore the same answer, word by word, against the clean prompt. This costs a single parallel forward pass, with no new generation. Tokens that became much more likely because the distractor was present are the ones the distractor caused. We push those specific tokens down and leave the rest of the failed answer alone. The approach turns failure into a targeted negative constraint learned in the model's weights, rather than a growing list of "the answer is not X" hints in the prompt. It also adds only about one teacher-forced pass per failed rollout to the cost of RL.
Core Claim
Our claim is that a token-level negative penalty, weighted by how much each token depends on injected irrelevant context, will produce three gains. It will yield more distraction robustness per GPU-hour than GRPO, than GRPO with confidence-weighted penalties on all-wrong groups, and than offline preference training on clean versus distracted pairs. It will do this without making the model ignore context that is actually helpful.
Concretely, let $x$ be a clean problem and $x_d$ the same problem with distractor sentences $d$ inserted. Let $y$ be a sampled answer that the verifier marks wrong, and let $\pi$ be the current policy. For each token $y_t$ we compute a distractor dependence score:
$$ s_t = \log \pi(y_t \mid x_d, y_{<t}) - \log \pi(y_t \mid x, y_{<t}). $$
$s_t$ is large when the distractor made the token more likely. Each failed rollout then receives a per-token negative advantage:
$$ A_t = -\lambda \, \frac{\max(s_t, 0)}{\frac{1}{T}\sum_{u} \max(s_u, 0) + \epsilon}. $$
Here $T$ is the answer length, $\lambda$ sets the penalty scale, and $\epsilon$ avoids division by zero. In groups that contain at least one correct answer, ordinary GRPO advantages are kept, and $A_t$ is added only to the wrong answers. In all-wrong groups, $A_t$ is the only signal. The clean-prompt rescoring is teacher-forced and fully parallel. Its cost is a small fraction of the autoregressive sampling it reuses.
Where This Sits
The proposal answers an open problem in RL efficiency. Don't Waste Mistakes (LENS) shows that all-wrong GRPO groups waste rollout compute. It fixes this with a sequence-level penalty scaled by the model's confidence in its wrong answer, evaluated on clean MATH problems. LENS is our closest prior work, and we keep its motivation. We differ in two ways. Our penalty is distributed across tokens according to what caused the error, rather than spread uniformly over the answer. Our target regime is noisy prompts, which LENS does not study. Not All Rollouts are Useful (PODS) provides a constraint we respect: it shows that dropping negative rollouts and training only on successes hurts. Our design therefore keeps negatives and makes them more informative.
The noise side motivates the target. How Is LLM Reasoning Distracted by Irrelevant Context? (GSM-DC) shows two things. Accuracy decays roughly as a power law in the number of distractors, and distractors cause both wrong reasoning paths and arithmetic slips. Its fix is a step-level reward model guiding tree search at inference time, which is slow. Thinking in a Crowd shows that explicit reasoning can make misleading context worse. Models build long rationalizations, so Qwen3-8B with thinking on falls below thinking off under misleading snippets. This suggests that the distractor-dependent tokens are often concentrated in identifiable spans, which our score is meant to find. Adaptive Distraction and Breaking Focus constrain the design in two ways. They show that "ignore irrelevant information" prompting gives almost no protection. They also show that offline preference training (DPO) on clean versus distracted pairs recovers much of the lost robustness, which makes DPO a baseline we must beat. AbstRaL gets distraction robustness from GRPO with a symbolic-distance reward. That reward requires oracle abstractions generated by a 70B model; our signal requires only the clean prompt we already have.
The negative-feedback literature both supplies the idea and exposes a tension. Progressive Rectification Prompting and Wrong-of-Thought feed failed answers back into the prompt as negation hints. These methods work well at inference time; PRP reaches 93.1% on GSM-IC2-1K. But they lengthen the prompt with exactly the kind of non-essential text that the distraction work shows is harmful, and they pay for several rounds of calls on every test question. We move the negative constraint out of the prompt and into training. At test time the model answers in one pass with no hint list. NLPO shows that constraining the policy's token choices stabilizes RL. Its mask is based only on token probability. Ours is informed by context and conditioned on failure, and it is applied as a gradient rather than a hard mask, so exploration is not truncated.
Research Program
We will build the method on a standard GRPO implementation and train small open models: Qwen2.5-0.5B and 1.5B, plus Qwen2.5-3B if compute allows. Training data comes from GSM8K training problems with injected distractors. We use two kinds of distractors. The first is generated by the GSM-DC procedure, which adds controlled irrelevant nodes to the problem's dependency graph. The second is narrative and misleading sentences written in the style of GSM-IC. Every distracted training prompt keeps a pointer to its clean source. This is the only extra supervision the method needs. It is free because we create the distractors ourselves.
The analysis has three parts. First, we measure how often GRPO groups are all-wrong on distracted prompts, and how that rate changes over training. This quantifies the compute that standard GRPO wastes in this regime, which the file does not report. Second, we validate the dependence score $s_t$ as an attribution. On GSM-DC, the dependency graph tells us which numbers belong to distractors. We check whether high-$s_t$ tokens coincide with tokens that copy or operate on distractor quantities, and compare against random and uniform attribution. Third, we train with the full objective and measure robustness, clean accuracy, helpful-context use, answer length under misleading context, and wall-clock and GPU-hour cost.
At test time, real distractors are not marked. The method uses distractor labels only during training, so this is fine for evaluation. As a secondary study we will test an unlabeled variant. It rescores failed answers with each prompt sentence removed in turn and attributes blame to the sentence with the largest effect. This variant is more expensive, so we report its cost separately.
Baselines
The naive reference point is the base instruction model with chain-of-thought prompting and no training. We also report the same model given an explicit "ignore irrelevant information" instruction. This variant is expected to help little, per Adaptive Distraction, and is included so that every reader sees where the floor is.
The main RL baseline is plain GRPO trained on the identical mixture of clean and distracted prompts. It uses the same number of rollouts per prompt, the same steps, and the same learning-rate sweep. This tests whether data augmentation alone explains any robustness gain. To beat it, our method must reach higher accuracy on the distraction benchmarks at matched GPU-hours. The rescoring pass is charged to our method, so GRPO is given the equivalent extra training steps.
The strongest competitor is LENS (Don't Waste Mistakes). We reimplement its confidence-weighted negative-group penalty inside the same GRPO code and train it on the same distracted data with the same tuning budget. It is the most direct rival because it already turns all-wrong groups into gradient. If our token attribution adds nothing beyond "penalize failures harder," LENS will match us. The file reports only that LENS "consistently outperforms the GRPO baseline across all evaluated metrics (Pass@k) on the MATH benchmark," with the largest gains on Levels 4–5. We found no reported LENS result on Lacuna for GSM-DC, GSM-Plus Distract, GSM-IC, or SciAux. We will therefore produce its numbers ourselves, and beating it means a statistically reliable gain across three seeds on distraction benchmarks with no loss on clean MATH-500.
The offline competitor is DPO on clean-versus-distracted answer pairs, following Breaking Focus. Pairs are built from the same source problems, and pair generation is counted against DPO's budget. That paper reports post-training DPO improving perturbed accuracy by "nearly 50% in some smaller models." That figure is on their own perturbation suite, so we treat it as indicative rather than directly comparable. We also report AbstRaL's figure, where Qwen2.5-0.5B accuracy on varied GSM-Symbolic problems rose from 34.0% to 44.6% (AbstRaL). This places our 0.5B results against an oracle-assisted RL method. For inference-time context, we cite PRP's 93.1% versus 70.7% for chain-of-thought on GSM-IC2-1K (Progressive Rectification Prompting). That result uses a much larger model and about four calls per question, so it is a reference, not a matched comparison.
Three ablations isolate the key ingredient, and all keep the total penalty mass per rollout identical to the full method. The first replaces $s_t$ with random weights. The second spreads the penalty uniformly across tokens, which is close to LENS without confidence weighting. The third computes $s_t$ against a different, unrelated distractor instead of the clean prompt. If the full method does not beat the first two, the attribution is not doing the work.
Why not just…?
Why not just run LENS on distracted data? This is the real threat, and we treat it as the central comparison rather than arguing it away. LENS already recovers gradient from all-wrong groups, and with distracted training data it may learn much of the same robustness. Our hypothesis is that a uniform sequence penalty also pushes down correct intermediate steps in a failed answer. Many answers fail on a single copied distractor number after several right steps, so this collateral damage is a real cost. The uniform-penalty ablation and the head-to-head LENS run settle the question. If neither is reliably worse than our method on GSM-DC path accuracy and GSM-Plus Distract, we will report that token attribution is unnecessary.
Why not just do offline DPO on clean versus distracted pairs? Breaking Focus shows this works, so it is a serious baseline. Its preferences are fixed before training. It cannot target the errors the current policy is actually making, and it needs pre-generated adversarial pairs. Our method gets its signal from the policy's own failures. We will compare at equal total compute, including pair generation for DPO. We will also test on distractor types held out from training, where we expect on-policy correction to generalize better. If DPO matches us there, the cheaper offline route wins.
Why not just strip irrelevant context at inference with a summarizer or path pruning? Less is More and Superposition Prompting show that summarizing or pruning context can raise accuracy. The file notes, however, that aggressive pruning risks discarding subtle, distributed constraints. Pruning also adds a stage to every query and does nothing for misleading context that looks relevant. We include a summarize-then-solve baseline using the same base model. On SciAux, the helpful-context condition tests directly whether pruning, or our training, throws away useful information.
Proposed Experiments
We evaluate on four benchmarks. GSM-DC lets us vary distractor count and separate step accuracy from path accuracy. For reference, Grok-3-Beta step accuracy falls from 43% with one distractor to 19% with fifteen on 5-step problems (GSM-DC), and we will plot whether our method flattens the power-law decay. On GSM-Plus we report the Distract category and the Performance Drop Rate (GSM-Plus), where open models show drop rates of 30–40%. We also use GSM-IC2-1K and GSM-ICM-1K; these derive from GSM8K, so we exclude any problem whose source overlaps our training split. The fourth is SciAux (SciAux Card), which covers a domain outside math and separates helpful, irrelevant, and misleading context. Clean MATH-500 and GSM8K test accuracy check that robustness does not cost capability.
On cost, we report three quantities. The first is the fraction of rollouts that contribute nonzero gradient. The second is the GPU-hours to reach GRPO's final distraction accuracy, which follows the speedup framing of PODS. The third is the measured overhead of the clean-prompt rescoring pass. We also measure answer length under misleading context to see whether the rationalization pattern from Thinking in a Crowd shrinks.
Three results would be decisive. The first is that the full method beats both LENS and the uniform-penalty ablation on GSM-DC path accuracy and GSM-Plus Distract at matched GPU-hours. The second is that high-$s_t$ tokens overlap with distractor-derived quantities on GSM-DC well above the random-attribution rate. If this fails, the mechanism is not what we claim, even if accuracy improves. The third is that accuracy in SciAux's helpful-context condition does not fall below GRPO's. A drop there would mean the model has learned to ignore context in general, the over-pruning failure the file warns about, and we would count it against the method.
References
- Don’t Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
- Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
- How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
- Get an A in Math: Progressive Rectification Prompting
- Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers