Reward models are the learned scorers that tell a language model which of its answers people would prefer. They are increasingly used to judge answers grounded in retrieved documents. A large body of work makes these scorers "uncertainty-aware" so that a training loop, or a human reviewer, can be pointed at the cases where the score is least trustworthy. Separately, it is well documented that language models are derailed by plausible but irrelevant text in their input. This note puts the two findings together. A reward model reading a cluttered context may be wrong and confident at the same time. If so, the uncertainty signals people already use to decide what a human should check will skip exactly these errors. The human budget will go to the wrong items.
We propose to test this directly and to fix it with a cheap signal we call context sensitivity. Context sensitivity is how much a single reward model's score for a fixed answer moves when the surrounding retrieved passages are dropped, reordered, or swapped. We use this signal, alongside standard uncertainty, to choose which examples a human labels. The claim is specific. On noisy-context benchmarks, choosing labels by context sensitivity should reduce reward-model errors more per human label than choosing them by ensemble disagreement, by embedding-covariance uncertainty, or at random. It should also do better than simply filtering the context before scoring.
Core Claim and Why It Is Not Already Done
Existing uncertainty methods detect inputs that look unfamiliar to the reward model. Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation measures how far a sample's final-layer features fall from the training feature covariance. It uses this to penalize the policy (ADVPO), and it suggests, without testing, that the same signal could drive active labeling. Uncertainty-aware Reward Model adds a Gaussian output head for noise in the labels themselves and uses seed-ensemble spread for the model's lack of knowledge. Both were evaluated on short, clean prompts. Neither asks what happens when the prompt is familiar in form but padded with distracting passages.
Contextual distraction is a different kind of failure. Breaking Focus shows that coherent distractors drop GPT-4o-mini from 85.7% to 22.0% on CommonsenseQA. The models answer correctly once the distractors are removed, so the failure is about attention and reasoning, not missing knowledge. There is no reason to expect such inputs to look out-of-distribution in feature space, since retrieved text is the normal input of a retrieval-augmented reward model. RAG-Reward motivates the concern. It reports that reward models scoring above 90% on RewardBench fall below 80% on its long-context OpenGenAlign test set, and it trains a better point-estimate reward model. It does not measure whether the remaining errors are flagged by any uncertainty signal, and it has no human escalation path.
The new contribution has three parts. The first is a measurement: when distraction causes a reward error, do existing uncertainty signals rise or not? The second is a signal aimed at that failure. The third is an evaluation of that signal where it matters most, deciding which examples a scarce human labels.
The Signal
Let $x$ be a query, $C = \{c_1,\dots,c_n\}$ the retrieved passages, and $y$ a candidate answer. Let $r(x, C, y)$ be the reward model's score. Draw $K$ perturbed contexts $C^{(k)}$. Each one drops a random subset of passages not cited by $y$, shuffles passage order, or replaces one uncited passage with a topical but irrelevant passage from the same corpus. Context sensitivity is the spread of the score across these variants:
$$s(x, C, y) = \mathrm{Std}_k \big[ r(x, C^{(k)}, y) \big].$$
The idea is that a judgment based on the evidence should not change when evidence the answer does not use is perturbed. A large $s$ means the score depends on irrelevant material.
For a preference pair, we label the pair by the larger of its two answers' $s$ values, or by the change in the reward margin, whichever proves more predictive on a development split. The cost is $K$ extra forward passes of one model. It needs no extra training, unlike an ensemble, which needs $K$ separately trained models.
The signal does not replace epistemic uncertainty; it is meant to cover a different set of errors. We will therefore also report a combined routing score, the rank-average of $s$ and the covariance metric from the lightweight-uncertainty paper.
Research Program
The first stage is a diagnostic stress test for reward models. We start from OpenGenAlign preference pairs, which cover long-context QA from WebGLM, Yelp data-to-text, and XSum summarization. We also use UDA question–document pairs (UDA), whose real PDF and HTML parsing noise is a natural source of distraction. We add two kinds of distractors. One is retrieved but irrelevant passages. The other is adversarial distractors produced with the error-guided tree search of Breaking Focus, run against the reward model rather than a reader.
Automated perturbations are often invalid. Evaluating adversarial attacks against multiple fact verification systems found only 34.0% human-judged correctness for automatic paraphrase attacks. We will therefore have annotators audit a stratified sample to confirm that each distractor does not change which answer should be preferred. Only distractor families that pass this audit are kept.
On the validated set we measure the drop in preference accuracy. For the pairs that flip, we compare the rise in each uncertainty signal against flips caused by ordinary out-of-domain prompts. The central diagnostic is the area under the ROC curve for detecting flipped pairs, computed per signal. If covariance or ensemble spread already detects distraction flips well, that is a publishable negative result, and the active-labeling stage collapses to "use existing uncertainty." We would report that rather than push $s$.
The second stage is budgeted active labeling. Each round, a routing rule selects $B$ pairs from an unlabeled pool of noisy-context pairs. Those pairs are labeled, and the reward model is updated. We compare routing rules at equal label counts.
Real humans are the scarce resource being studied, so we cannot buy full learning curves with them. Curves are therefore run with a simulated labeler: the reward model's preference on the clean-context version of the pair, backed by an o3-mini majority-vote judge, the pipeline RAG-Reward reports at 81% agreement with professional journalists. The final comparison of the two best rules is repeated with real annotators at a fixed budget of a few hundred labels. That scale is comparable to the under-400-label regime of Nano.
The third stage tests whether better reward-model labels translate into better policies. We train a Llama-3.1-8B-Instruct reader with PPO against each resulting reward model. The policy objective subtracts a penalty $\beta \cdot s$ from the reward, so the policy is not rewarded for answers whose score depends on noise. We evaluate on UDA, NQ, TriviaQA, and HotpotQA with distractor passages added.
Baselines
The naive reference point routes labels uniformly at random and scores with the off-the-shelf reward model and no penalty. It shows how much any routing rule buys.
The strongest existing uncertainty method for this question is the single-model covariance metric with ADVPO from Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation. It is the strongest because it matches the low cost of our signal, and its authors already proposed it for exactly this active-labeling use. Its reported result is a 75% win rate over standard PPO on TL;DR, judged by GPT-4 and humans. We found no reported active-labeling result for it on Lacuna, and none for any method on noisy-document distributions. We run it on the same base reward model, the same pool, and the same label budgets, and tune its threshold on the same development split. Beating it means a lower reward-model error at equal labels on distraction pairs, without being worse on clean pairs, across three seeds.
A five-member seed ensemble in the style of URME (Uncertainty-aware Reward Model) is the accuracy upper reference for epistemic uncertainty. The related single model URM(S) reports 92.9 on RewardBench. We report its compute cost separately so the comparison is honest about cost.
A filter-then-score baseline passes the context through a distraction-aware selector before the reward model sees it. We use the passage-band selection of LDAR and the training-free query-likelihood reweighting of DAS. The selector is tuned on the same development data.
For absolute reward accuracy, we also include the RAG-Reward model itself, which reports about 86% on its held-out RAG benchmarks.
The ablation that isolates the key ingredient keeps the $K$ forward passes but replaces context perturbation with perturbation of the response, by paraphrase or dropout noise. If that matches $s$, the gain comes from test-time sampling in general, not from probing context dependence.
Why not just…?
Why not just filter the context before the reward model scores it? This is the most serious threat. If LDAR- or DAS-style filtering makes the reward model's errors on noisy inputs go away, no uncertainty signal or human routing is needed. There are reasons to doubt it fully works. LDAR selects passage bands from retriever score patterns without reading the text, and it cannot notice when every passage in the band is misleading. A reward model that checks attribution also needs the evidence that filtering might remove. Still, this is an empirical question, and filter-then-score is a headline baseline. If it closes most of the accuracy gap, we narrow the claim to the residual errors that survive filtering and test whether $s$ computed after filtering still routes labels better.
Why not just tell the model to ignore irrelevant information? Breaking Focus tested exactly this instruction and found it ineffective against its adaptive distractors. We still include an instruction-prefixed reward model as a cheap baseline, because reward models were not part of that test.
Why not just use an LLM judge instead of humans? We do use one, as the simulated labeler and as a cheap first tier. However, RAG-Reward reports that GPT-4o agreed poorly with professional humans on long-context judgments, and even the o3-mini voting pipeline reaches only 81%. HILA shows that models sharing pretraining blind spots fail together, and that human deferral was what lifted AMC accuracy to 35.83%. We will report how often the judge disagrees with real annotators on high-$s$ pairs specifically. If the judge is reliable there, the human tier can shrink, which is itself a useful result.
Why not just use a reward-model ensemble? It is included as a baseline. The open question is whether ensemble members trained on the same data share the same distraction failure. If they do, their disagreement stays low exactly when they are all fooled.
Decisive Checks
The proposal stands or falls on two results.
The first is the diagnostic. Among pairs whose preference flips under validated distractors, existing uncertainty signals should separate them from correct pairs much worse than they separate out-of-domain flips, while $s$ separates them substantially better.
The second is the labeling curve. At fixed budgets, routing by $s$ or by the combined score should reach a given reward-model accuracy on noisy-context pairs with fewer human labels than both covariance routing and filter-then-score. The confirmation with real annotators should agree in direction.
A downstream check follows. PPO with the $s$ penalty should improve reader accuracy on UDA and distractor-augmented QA over the covariance-penalized ADVPO setup, at matched KL divergence from the starting policy.
If the diagnostic shows that existing uncertainty already catches distraction errors, we report that and stop, since that would settle Gap 1 in the negative.
References
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation
- Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- RAG-Reward: Optimizing RAG with Reward Modeling and RLHF
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Evaluating adversarial attacks against multiple fact verification systems
- Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning