Language models often know how to solve a math word problem but fail once a few irrelevant sentences are added to it. They pull the extra numbers into the calculation, or they write long chains of reasoning that justify a misleading premise. Reinforcement learning with checkable rewards should be able to train this habit away, but the most common recipe, Group Relative Policy Optimization (GRPO), has a blind spot here. GRPO scores each sampled answer against the other answers drawn for the same prompt. On a distractor-laden prompt, all of those samples tend to swallow the same distractor. They then fail together, the reward spread within the group is zero, and the update carries no learning signal.
We propose to make the sampled group diverse in a specific dimension: which premises of the problem each attempt actually uses. This replaces diversity in wording. Concretely, we track which stated quantities each partial solution has used. We branch generation at the moments the model is about to commit to a premise. From an oversampled pool, we keep a subset whose premise-usage patterns differ and whose rewards spread widely. The reward stays outcome-only. Only the composition of the group changes. Our bet is that this recovers learning signal exactly on the prompts where distraction occurs, and that the resulting policy degrades much more slowly as distractors are added.
Core Claim
On prompts containing irrelevant context, GRPO groups collapse in premise space, not in token space. Two rollouts that differ in many words but both use the distractor quantity are, for learning purposes, the same rollout. We claim three things. Group diversity measured by premise-usage signatures predicts within-group reward variance far better than string edit distance does. Enforcing signature diversity during rollout generation cuts the fraction of zero-variance groups on noisy prompts. The trained policy shows flatter accuracy decay on controlled-distractor benchmarks, with no loss on clean ones.
The claim is new relative to two literatures that have not met. Diversity-aware GRPO samplers all assume clean prompts and measure diversity on surface text or rewards. Lookahead Tree-Based Rollouts (LATR) branches at high-uncertainty tokens and prunes branches by normalized edit distance. S2L-PO mixes in rollouts from a smaller model of the same family. PODS down-samples a large rollout pool to the highest- and lowest-reward samples. GRPO methods that target distraction, by contrast, change the reward or the reasoning format but keep naive independent sampling. EIT rewards resisting injected bias cues. AbstRaL forces translation into symbolic templates distilled from a 70B teacher. We change neither the reward nor the format. We change which trajectories the group is built from, using a diversity notion tied to the failure mode.
Why Premise Space
Three findings motivate the design. GSM-DC separates logical path accuracy from arithmetic step accuracy under controlled distractors. It shows that errors grow steadily with distractor count; Grok-3-Beta step accuracy falls from 43% with one distractor to 19% with fifteen. That points to premise selection, not computation, as the failure point. Breaking Focus shows the knowledge is present but the filtering ability is not, and that "ignore irrelevant information" prompts do not help. If the model can solve the clean problem, then somewhere in its distribution there are trajectories that skip the distractor. The sampler's job is to find them. Thinking in a Crowd shows that misleading snippets make explicit reasoning harmful: Qwen3-8B reaches 77.03% with thinking ON, 74.87% with it OFF, against 96.22% clean. That is the setting where whole groups rationalize the same wrong premise and GRPO stalls.
One finding constrains the design. LATR's own limitation section, as summarized in the file, notes that edit distance can misjudge semantic diversity. We take that as the reason to swap the metric while reusing LATR's branching machinery.
Research Program
Premise signatures. Training problems are built so that every premise is identifiable. For GSM-DC-style problems, the generator's dependency graph labels each sentence relevant or irrelevant. For GSM8K problems with injected distractors, we write distractor sentences whose numeric values do not occur elsewhere in the problem. A trajectory's premise signature $s(o)$ is the set of premise indices whose quantities appear in the trajectory's arithmetic. We extract it by matching numbers and named entities. This is cheap and needs no model call. It is a proxy: a model could use a value without restating it. So we validate the proxy on a held-out sample against manual annotation of a few hundred trajectories before relying on it.
Premise-crossroad branching. LATR branches where next-token entropy is high. We instead branch at the first position where a partial trajectory is about to introduce a premise not yet in its signature. From that point we draw several short continuations (LATR's 20–50 token lookahead). We keep the continuations whose signatures differ from their siblings, measured by Jaccard distance, and drop the rest. Branches are drawn from the unmodified policy, so the rollouts stay on-policy and the standard GRPO ratio applies. As an ablation only, we test a stronger off-policy intervention: briefly masking the distractor's number tokens at a crossroad, with importance correction.
Signature-aware group selection. We generate a pool of $n$ rollouts and keep $m<n$ for the update, so the backpropagation cost matches standard GRPO. Following PODS, we first reserve the highest- and lowest-reward rollouts. The remaining slots are filled greedily. Each step adds the rollout that maximizes
$$\Delta(o) = \min_{o' \in S} \big(1 - J(s(o), s(o'))\big),$$
where $S$ is the set already selected and $J$ is Jaccard similarity of premise signatures. This is non-maximum suppression in premise space, the same logic DICE uses to keep rare driving maneuvers alive among predicted vehicle paths.
When the whole pool has reward zero, the procedure still keeps the most premise-diverse failures. This adds no gradient by itself. It does tell us whether branching produced any distractor-free attempts, which we log as a diagnostic.
Annealing. Like LATR, we anneal branching toward plain sampling late in training, so the test-time sampling distribution matches training.
Baselines
The naive reference is standard GRPO trained on exactly the same distractor-augmented data, with independent temperature-1 sampling and group size $m$. It isolates whether any sampler change matters once the data already contain distractors. Many reviewers will expect this baseline alone to close most of the gap. Beating it means a statistically reliable gain on GSM-DC path accuracy at high distractor counts (eight or more), over at least three seeds. We have found no reported GRPO-on-GSM-DC result on Lacuna, so this baseline must be run by us.
The strongest existing sampler is LATR, run on the same noisy training data with its edit-distance pruning. LATR reports raising OlympiadBench Pass@1 from 28.1% to 30.4%, and an average Pass@1 gain of 4.2%, on clean benchmarks (LATR). It is the closest competitor because our method keeps its branching and lookahead machinery and changes only the branch trigger and the diversity metric. We also run PODS with the same $n$ and $m$. It tests whether reward-variance down-sampling alone already recovers signal; PODS reports reaching peak GSM8K accuracy at least 1.7x faster than standard GRPO (PODS).
For distraction robustness, the strongest existing trained method on our evaluation domain is AbstRaL. It reports Qwen2.5-0.5B on varied GSM-Symbolic problems rising from 34.0% to 44.6% (AbstRaL). It also reports beating chain-of-thought on the GSM-Plus distractor subset, though no exact number is given (no reported result found on Lacuna for AbstRaL's GSM-Plus Distract accuracy). We compare at the 0.5B scale on GSM-Symbolic so this number is directly comparable. Matching AbstRaL without its teacher-distilled symbolic annotations would count as success; exceeding it would be a strong result.
Fairness rules apply to every method. All use the same base models (Qwen2.5-0.5B, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct), the same training prompts, and the same number of rollouts backpropagated per prompt. Generated-token budgets are matched by giving the baselines larger independent pools. Each method gets the same hyperparameter sweep of six configurations.
Two ablations isolate the key ingredient. The first keeps our branching trigger but prunes by edit distance. The second keeps LATR's entropy trigger but selects by premise signature. If only the full combination helps, the contribution is the pairing. If the signature selection alone accounts for the gain, the method simplifies to a cheap down-sampling rule, which we would report as the headline.
Why not just…?
Why not just raise the sampling temperature? S2L-PO and LATR both report that token-level noise yields incoherent steps or trivial synonym swaps rather than different strategies. We still include temperatures 1.0, 1.2 and 1.4 in the standard-GRPO sweep. We report each one's zero-variance group rate on noisy prompts, so the claim is checked in this specific regime rather than assumed.
Why not just reward the model for not using distractor premises? The same signature we use for selection could serve as a process reward. This is a real threat, because it is simpler and directly targets the behavior. Our concerns are concrete. The signature is a number-matching proxy, and a model can learn to avoid restating distractor values while still being swayed by them. EIT documents an analogous "performative independence" pattern, in which models claim to ignore cues and then use them (EIT). A shaped reward also cannot help in all-fail groups that never produced a clean trajectory. We run this baseline at matched compute and also combine it with our sampler. If shaped reward alone matches us on GSM-DC and on the transfer sets, which have no signature labels, we will report that the selection mechanism is unnecessary.
Why not just strip distractors with a filter before the model reads the prompt? Ext2Gen found that separate compressors underperform joint extraction and generation under heavy noise. Breaking Focus found that instruction-based filtering fails. A filter also does nothing for SciAux-style misleading snippets that are on topic. We include a pipeline in which an LLM first rewrites the prompt to remove irrelevant sentences, and evaluate it on the same test sets.
Why not just use PODS on noisy data? PODS keeps high- and low-reward extremes, which helps whenever some rollout succeeds. Our hypothesis is that the distraction failure is precisely the all-fail group, where PODS has nothing to select. The diagnostic below measures how often that case occurs. If it is rare, PODS is sufficient and our branching adds only cost, roughly the 10% generation slowdown LATR reports.
Proposed Experiments
Training data combines GSM-DC problems generated with one to ten distractors and GSM8K problems with injected distractor sentences. In-distribution evaluation uses GSM-DC step and path accuracy across one to fifteen distractors. Fitting the decay curve per method tests whether training changes its slope, not just its intercept. Transfer evaluation uses the GSM-Plus distractor-insertion and critical-thinking subsets, reporting performance drop rate against GPT-4's 8.23% and the 30–40% of open models, plus GSM-Symbolic for the AbstRaL comparison. Out-of-domain transfer uses SciAux misleading-snippet conditions with thinking enabled. The question there is whether the gap between 77.03% and the 96.22% clean baseline narrows, and whether chain length on misleading prompts shrinks, which would indicate less rationalization. Clean GSM8K and MATH-500 guard against regression.
The first decisive check is diagnostic and runs before any training. For the base model on distractor prompts, we compute the per-group correlation between reward variance and each diversity measure: edit distance versus premise-signature distance. If edit distance predicts variance as well as signatures do, the core premise is wrong and LATR suffices. The second check is the zero-variance group rate during training for each sampler. Our method should reduce it substantially relative to both standard GRPO and PODS, and the reduction should track the gain in high-distractor path accuracy. The third check is the shaped-reward comparison on the transfer sets, which decides whether premise diversity is better used to choose rollouts or to score them.
References
- Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards
- Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
- Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
- Treat Bias as Noise: Training Bias-Robust LLM Reasoning via Reinforcement Learning
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
- How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- DICE: Diverse Diffusion Model with Scoring for Trajectory Prediction
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers