Catching Judges That Accept Rationalized Noise

Language models are increasingly graded by other language models. The grader, called an "LLM judge," reads a question, the material the model was given, and the model's step-by-step answer (its chain of thought), then decides whether the answer is good. This note targets one specific way that arrangement fails. When a solver is handed a misleading document, it often does not ignore it. It writes a long, fluent argument that builds on the false premise and lands on a wrong answer. We ask whether judges, especially judges that themselves reason step by step before scoring, are persuaded by exactly these arguments. We then propose a cheap way to train judges that are not persuaded, without paying humans to annotate reasoning steps.

The training data comes from controlled noise injection. Take multi-hop questions whose answers require reading supplied passages. Plant a passage that contradicts one of the needed facts, and record the solver's reasoning with and without it. Because we planted the passage, we know which reasoning traces were derailed and which step imported the false fact, so the labels cost nothing. We train the judge to trace each premise of the solver's argument back to the passage it came from, and to reject arguments resting on a passage that conflicts with the rest of the evidence. The claim is that this provenance-checking judge rejects rationalized noise far more reliably than current rationale-first judges, and that it does so in settings where the judge cannot simply re-solve the question itself.

Core Claim

Two findings motivate the work. Thinking in a Crowd introduced SciAux, a set of ScienceQA-derived questions paired with helpful, irrelevant, and misleading snippets. It showed that reasoning modes fall into a "rationalization trap." With two misleading snippets, Qwen3-8B with thinking enabled scores 77.03%, down from 96.22% without misleading context. Its longest outputs occur while it is reconciling the misleading information. Local Coherence or Global Validity? showed that outcome-rewarded training produces traces that read as locally coherent but are globally invalid. It also found that a GPT-4o judge agrees with human error tags only 57.8% of the time. Together these predict that the traces most in need of rejection are the ones best built to fool a judge that rewards fluency.

Meanwhile, the most effective trained judges are rationale-first. TRACT combines chain-of-thought with a squared-error loss on the score. Self-rationalization improves LLM as a fine-grained judge trains judges with preference optimization over their own rationales. Neither exposes the judge to noisy context inside the material it grades. Our claim has two parts. The diagnostic part: rationale-first judges accept noise-derived wrong traces at a substantially higher rate than they accept equally wrong traces that contain no planted noise. The constructive part: training on injection-labeled pairs, with a rationale that explicitly attributes premises to passages, closes most of that gap without hurting standard judging benchmarks.

Relation to Prior Work

Two kinds of diagnostic work show that judges hide what drives their verdicts, and our proposal moves from diagnosis to training. The Silent Judge shows that provenance and recency labels shift GPT-4o's pairwise verdicts by up to +30% on ELI5, while the judges' justifications never mention the cue (0% acknowledgment). That study proposes no fix and studies metadata labels, not misleading content inside the graded material. Counterfactual Simulation Training trains a task model's chain of thought so that a simulator can predict its behavior under counterfactual inputs. It gains 35 to 48 points of monitor G-mean on cue-based counterfactuals, but it has a documented blind spot for cues that push away from an answer, and it targets the solver rather than the judge. We borrow its central move, using counterfactual inputs whose effect we control as a free supervision signal, and apply it to the judge.

Two methods supply our training machinery: the TRACT dual loss with its self-generated second-stage rationales, and the self-rationalizing evaluator's preference pairs. Our contribution is the data and the rationale format, not a new loss.

Two lines of work constrain the design. Ext2Gen shows that making a generator extract supporting sentences before answering beats separate compression modules under noisy retrieval. It reports an average score of 0.557 versus 0.473 for default Llama-3.1-8B-Instruct. We take from this that grounding in specific passages is the right intermediate representation, and we apply it on the judging side. LaTRO and the Mixed SFT study show that reasoning can be learned without step labels, but both assume clean inputs and verifiable answers. Our setting deliberately removes the gold answer at judging time.

Research Program

Data construction starts from items whose answers cannot be recovered from memory. We take multi-hop HotPotQA items from the Ext2Gen alignment data, which already mixes in up to 25 irrelevant chunks, and the SciAux items with helpful snippets. We keep only items where a strong model answering with no passages ("closed-book") fails but succeeds given the gold passages. For each kept item, a frontier model rewrites one gold supporting passage into a fluent contradicting version, and we insert it alongside the original and the distractors.

Solvers (Qwen3-8B and Hunyuan-7B-Instruct, with thinking on and off) produce traces under clean and contaminated contexts. A contaminated trace is labeled "derailed" when its answer is wrong and it lexically or semantically cites the planted passage. The citing step is the labeled error location. Clean, correct traces form the positive side of each pair.

Length is a known confound, since derailed traces are the longest. We therefore length-match pairs, and we hold out a test split that deliberately pairs short derailed traces with long correct ones.

The judge is trained from Llama-3.1-8B-Instruct and from Mistral-7B so that results line up with SRE and TRACT. Its rationale format is fixed. It must list the premises the trace relies on, attribute each to a passage or to background knowledge, and flag premises whose source conflicts with other passages. Only then does it emit a score. Supervision for the attribution step comes from the injection record. Scores are trained with the TRACT objective, and rationales are refined with SRE-style preference pairs. In those pairs, the chosen rationale correctly names the planted passage and the rejected one does not, so no meta-judge is needed to curate them.

Baselines

The naive reference point is an off-the-shelf rationale-first judge (Llama-3.1-8B-Instruct, and GPT-4o as a frontier reference) prompted with the question, all passages, and the trace. It is run twice, once with chain-of-thought and once score-only. The comparison between those two runs is itself the first diagnostic.

The strongest existing methods are TRACT and the self-rationalizing evaluator, retrained on the same base models. TRACT reports an average Pearson correlation of 0.650 on Feedback Bench (versus 0.591 for Prometheus-2-7B) and 73.6% pairwise accuracy on RewardBench with Mistral-7B. The SRE paper reports gains of 3% to 9% over baselines on RewardBench and BiGGen Bench. Both are the right competitors because they are the best published judges that reason before scoring, and they are the judges our diagnostic predicts are most vulnerable. For a fair comparison, each is additionally fine-tuned on our exact training items and solver traces, with the same number of examples, epochs, and tuning budget. The only difference is that they receive outcome labels (derailed or not) instead of passage-attribution rationales. This makes the "same data, ordinary rationale" condition the key ablation isolating our ingredient.

No reported result exists on Lacuna for any judge on noise-contaminated reasoning traces, so the bar on the new test set is set by our own reproduction of these baselines. Beating them means a lower acceptance rate for derailed traces on the length-adversarial split, at equal or better acceptance of correct traces. Their reported Feedback Bench and RewardBench numbers are the floor we must not fall below.

Why not just…?

Why not just tell the judge to ignore misleading or irrelevant context? The Silent Judge evidence suggests that instructed judges produce content-focused justifications while still being swayed, and CST found prompting insufficient for faithfulness. Both results, however, concern metadata cues or solver reasoning, not planted content in graded traces. We therefore run this prompt as a baseline rather than dismissing it. If it closes most of the gap, the training contribution is moot and the paper becomes a diagnostic one.

Why not just have the judge solve the question itself and check agreement? On plain SciAux this would be a serious threat, because the base model already scores 96.22% without misleading context, so agreement checking would catch most derailments. This is exactly why we restrict to items where closed-book solving fails. We include a "judge solves with the same passages, then compares answers" baseline. If the judge is itself derailed by the planted passage, agreement checking inherits the error. Our experiments measure whether that happens, and on what fraction of items.

Why not just hide the context from the judge? A judge that sees only the question and trace cannot be distracted by passages. But on context-dependent items it also cannot tell a legitimate passage-derived premise from a planted one. We run this baseline and expect it to reject correct traces at a high rate. If it does not, that finding argues that judges should never see retrieved context, which would also be worth reporting.

Why not just train on outcome labels without the attribution rationale? This is the ablation described above, and it is the real threat. If outcome-labeled TRACT matches us, then the free labels, not the provenance rationale, carry the gain. We would report it that way.

Proposed Experiments and Decisive Checks

The first check is the diagnostic. On held-out contaminated pairs, does chain-of-thought increase a judge's acceptance of derailed traces relative to score-only judging, and does acceptance grow with trace length? A null result here would undercut the motivation, and we would report it before building anything.

The second check is the main comparison. We measure derailed-trace acceptance, pairwise accuracy on matched pairs, and passage-attribution accuracy, the last scored against the injection record. Each is reported on the length-adversarial split and on a split whose planted passages push away from the correct answer rather than toward a specific wrong one, which probes CST's dissuasion blind spot. We also track a content analogue of cue acknowledgment: how often the judge's rationale names the planted passage when it accepts a derailed trace.

The third check is transfer. We test injected-noise GSM8K word problems for a domain the judge was not trained on. We also report Feedback Bench Pearson and RewardBench accuracy, where we must not fall below TRACT's reported 0.650 and 73.6%. Finally, as an exploratory test of whether resistance to content noise carries over to metadata cues, we run the ELI5 and LitBench recency and provenance protocol from The Silent Judge.

The idea is supported if the trained judge cuts derailed-trace acceptance substantially below the outcome-only ablation on the length-adversarial split while matching the standard benchmarks. It is refuted if the outcome-only ablation or the agreement baseline performs as well.