Language models that solve a math problem correctly often fail on the same problem once one plausible but irrelevant sentence is added. Error-guided searches for such sentences cut accuracy by roughly 45% on average across frontier models, and telling the model to "ignore irrelevant information" does almost nothing (Breaking Focus, Adaptive Distraction). Offline reinforcement learning (RL) for LLMs takes a different approach. It learns, from fixed logs of reasoning attempts, a value head: a small network that scores how likely a partial chain of reasoning is to end in a correct answer. That score is then used to steer generation (OREO). A value head trained only on clean prompts, however, has never seen a distracted one. In offline-RL terms, distracted prompts are out-of-distribution states. Robotics benchmarks such as D4RL show what learned values do in such states: they hallucinate optimism, and the policy follows them into failure (D4RL).
This proposal treats a distractor as observation noise over a hidden "true problem" that we know at training time, because we inserted the distractor ourselves. We train the value head on distracted prompts using a pessimistic target. For each partial reasoning chain, we finish it twice, once with the distracted prompt and once with the clean prompt, and the target is the lower of the two success rates. A chain scores high only if it is sound with respect to the real problem and the policy can also carry it through under distraction. At test time this critic guides beam search on distracted inputs. The claim is that this privileged, pessimistic target buys robustness that ordinary data augmentation plus outcome rewards does not. If augmentation alone matches it, the idea fails, and the experiments are designed to show which is true.
Core Claim
We have $x$, a clean problem, and $\tilde x = x \oplus d$, the same problem with a distractor $d$ inserted. Let $h_t$ be a reasoning prefix produced by the policy from $\tilde x$, and let $\hat G(c, h_t)$ be the fraction of $K$ sampled continuations of $h_t$, conditioned on context $c$, that reach the correct final answer. The value head $V_\phi(\tilde x, h_t)$ is regressed toward
$$y(\tilde x, h_t) = \min\{\hat G(\tilde x, h_t),\ \hat G(x, h_t)\},$$
and this term is added to OREO's soft-Bellman objective. The two arguments do different jobs. The clean-context continuation $\hat G(x, h_t)$ asks whether the prefix is still recoverable once the distractor is removed. A prefix that has already imported the distractor's numbers or premises will usually fail here, even when it happens to reach the right answer under $\tilde x$. The distracted continuation $\hat G(\tilde x, h_t)$ asks whether the current policy will actually finish the job in the deployment context. Taking the minimum is a direct analogue of the in-sample worst-case relabeling used by the Adversarially Robust Decision Transformer. That method relabels offline trajectories with minimax returns, estimated by expectile regression, so that the policy does not trust optimistic returns under adversaries. Here the "adversary" is the distractor rather than an opponent's moves.
The specific, testable prediction concerns overestimation. A critic trained the OREO way, even on distracted data, should assign systematically higher values to distractor-contaminated prefixes than their realized success rates justify. The min-target critic should shrink that gap. The gap should be measured on distractors that were generated by search against the final trained system.
Relation to Prior Work
Two papers motivate the gap. OREO trains an explicit value head jointly with the LLM via a soft Bellman equation and uses it for value-guided beam search. It reports 52.5% versus 47.5% for SFT on MATH with a 1.5B Qwen backbone, and 79.1% versus 69.4% for DPO on unseen ALFWorld rooms. It is evaluated only on clean inputs and uses no pessimism, so it is our base method, and our change is confined to the value target. The distraction papers (Breaking Focus, Adaptive Distraction) supply the threat model, a tree search that produces single-sentence semantic perturbations. They also supply the observation that models are best broken by distractors they generate against themselves. That observation forces our test-time attack to target the final system, critic included, rather than the base model.
Two lines of work supply the method and also constrain it. Contextual Conservative Q-Learning adds Gaussian noise to robot states and uses an inverse dynamics model to keep values high only for predictable transitions. It shows that observation noise can be handled inside the value objective. Its noise is continuous and its supervisor is a learned dynamics model. Ours is semantic, and our supervisor is the exactly known clean problem, which removes the compounding model error the file flags for dynamics-model methods. Sparse Q-Learning shows that in-sample value learning stays stable when datasets mix expert and random transitions, while competitors degrade. This argues for fitting targets only on prefixes the policy actually produced, never on synthetic out-of-sample steps, and we follow that.
Some work constrains the design. RoiRL gets strong self-supervised gains, 60.4% on MATH500 with single-sample decoding, by rewarding majority-vote answers. Under distraction, though, the majority converges on lure answers, and ConsisEval attributes many easy-problem failures to redundant context. So we use ground-truth correctness rather than self-consistency to compute $\hat G$. Divide-and-Conquer prompting shields attention by splitting tasks into independent parts, but its authors limit it to weakly coupled sub-tasks, which excludes most sequential math chains and ALFWorld episodes.
Research Program
The first component is a distraction corpus with known clean counterparts. For GSM8K and MATH training problems, we generate distractors from three sources: static template sentences, sentences from the tree-search procedure of Adaptive Distraction run against the base policy, and "redundant information" edits in the style ConsisEval identifies. For ALFWorld (ALFWorld) we inject irrelevant but plausible objects and descriptions into text observations. Because each distractor is inserted, the clean state is always available during training.
The second component is target computation. We sample policy rollouts from distracted prompts, cut them at step boundaries, and finish each prefix $K$ times under both contexts. In ALFWorld, "clean continuation" means replaying the episode prefix with distractor text removed from the observations.
The third component is training and inference. We run OREO's joint policy and value training with the added regression term, then decode with value-guided beam search exactly as OREO does.
The fourth component is analysis. For each critic, we bin prefixes by predicted value and compare against realized success on distracted inputs. We do this separately for prefixes that reference distractor content and prefixes that do not, which gives an LLM analogue of the extrapolation-error curves used in D4RL.
Baselines
The naive reference is the SFT policy with greedy decoding, plus the same model given an explicit "ignore irrelevant details" instruction. The latter is included because it is the first thing any practitioner tries, and the prior evidence that it gives almost no benefit should replicate.
The strongest existing distraction defense is DPO on clean versus distracted reasoning pairs, following Adaptive Distraction. That paper reports robustness gains of "nearly 50% in some smaller models" on perturbed inputs. We build its pairs from the same distraction corpus and the same rollouts we use, so it sees identical prompts.
The strongest existing offline-RL method is OREO, in two forms: as published on clean data, and retrained on the clean-plus-distracted mixture with ordinary outcome rewards. The clean-data reference numbers are 52.5% on MATH (Qwen-1.5B) and 79.1% on unseen ALFWorld rooms (OREO). No reported result was found on Lacuna for OREO or DPO on distracted MATH, GSM8K, or ALFWorld, so these numbers come from our runs. Augmented OREO is the baseline that matters most.
Fairness rests on generated-token budget, because our method spends $2K$ continuations per prefix. Augmented OREO and DPO get the same total generation budget, spent as additional distracted rollouts and pairs. All methods share the same backbone, the same number of gradient steps, and an equal hyperparameter sweep on a held-out validation split of statically distracted problems. That split is never drawn from the adaptive test distractors.
Two ablations isolate the key ingredient: using only $\hat G(\tilde x, h_t)$ and using only $\hat G(x, h_t)$. Beating the strongest baseline means three things. Accuracy under adaptive distraction must be higher than augmented OREO's at matched token budget. The improvement must exceed seed variation across three seeds. Clean accuracy must not fall more than a point below clean OREO.
Why not just…?
Why not just retrain OREO on distracted prompts? This is the real threat. If distraction is only a coverage gap, augmentation closes it, and our min-target adds cost for nothing. Our argument against it is mechanistic, not yet empirical. With outcome-only rewards, a contaminated prefix that reaches the right answer by luck is rewarded, and distracted success rates near 22% (GPT-4o-mini on CommonsenseQA fell from 85.7% to 22.0% in Breaking Focus) give a sparse, noisy signal. The calibration analysis and the matched-budget comparison settle the question directly. If augmented OREO shows no residual overestimation on contaminated prefixes, we report that pessimism is unnecessary.
Why not just use DPO on clean versus distracted pairs? It is simpler and has already helped small models. It assigns credit at the sequence level, however, and OREO shows DPO trailing explicit value learning by about ten points on ALFWorld. That gap is plausibly larger when the error is a single step that imports a distractor fact. DPO also gives no test-time signal for pruning a contaminated branch. The head-to-head comparison, broken down by chain length, shows whether step-level credit matters here.
Why not just strip distractors first, with a filter or by rewriting the prompt? A PRCA-style adapter compresses retrieved text well. It is tied to the generator it was trained with, and it cannot see distractors that are woven into the problem statement or that arrive mid-episode in ALFWorld observations. We include a "rewrite the problem, then solve it" variant using the policy itself. If adaptive distractors pass through the rewrite, as the prompting failures in Adaptive Distraction suggest they will, the defense has to live in the reasoning process itself.
Proposed Experiments
The main test uses GSM8K, MATH, and MATH-500 under three conditions: clean, static distractors, and adaptive distractors. The adaptive distractors are regenerated by tree search against each final trained system, with its value-guided search included, at an equal search budget per system. We report accuracy and the robustness gap (clean minus adaptive). ConsisEval's 298 GSM8K-derived pairs measure the Consistency Score $P(\text{easy}\mid\text{hard})$ with and without redundant-context edits. ALFWorld unseen rooms with injected distractor observations measure success rate.
The decisive checks are three. First, the overestimation gap on contaminated prefixes must be large for OREO-family critics and much smaller for ours; if it is small for every critic, the D4RL analogy fails. Second, the min-target must beat both single-context ablations, which shows the pessimism and not merely privileged information is doing the work. Third, our method must beat augmented OREO at matched tokens on adaptive distractors.
A small secondary experiment returns to D4RL. On Hopper, HalfCheetah, and Walker2d with Gaussian observation noise, we apply the same clean-versus-noisy min-target to a CQL critic and compare it against C-CQL. This tests whether privileged pessimism helps in general under known-noise observation corruption, or only in language. No per-task values for C-CQL's noisy settings were found on Lacuna, so both methods would be rerun under one protocol.
References
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Offline Reinforcement Learning for LLM Multi-Step Reasoning
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.
- Adversarially Robust Decision Transformer
- Contextual Conservative Q-Learning for Offline Reinforcement Learning
- Sparse Q-Learning: Offline Reinforcement Learning with Implicit Value Regularization
- RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter