Teaching Step Judges to Ignore Style Using Multi-Hop Decompositions

Language models are now routinely used as judges, grading other models' answers and, more and more, grading each intermediate step of a reasoning chain. These judges are known to reward the wrong things: longer answers, confident tone, an "EXPERT" label, a recent date. In The Silent Judge, such cues shifted GPT-4o's verdicts by up to 30%, and its written explanations mentioned the cue 0% of the time. Because the bias never shows up in the judge's stated reasons, you cannot fix it by reading those reasons. You have to measure and correct what the judge does when only the cue changes.

We propose to get the needed supervision almost for free from multi-hop question answering. These are questions like "In what city was the director of Inception born?", which need a chain of lookups where each step depends on the one before. Datasets such as MuSiQue are built from chained single-hop questions with known answers, so every step of a reasoning chain can be checked automatically: did it name the correct intermediate entity (here, Christopher Nolan), and is that entity supported by the gold paragraph? This makes it possible to build a factorial set of step variants. Some keep the content fixed and change only style (padding, confident tone, provenance labels, citation formatting) at controlled intensities. Others keep style fixed and swap in a wrong intermediate entity. We train a small set of learned input vectors, called a continuous or soft prompt, that sits in front of a frozen judge. The training objective makes the judge's step score flat as style intensity rises and sharply sensitive to substance. We then test whether this invariance carries over to bias types the prompt never saw and whether it improves multi-hop reasoning when the judge guides the search.

Core Claim

Gold multi-hop decompositions give a cheap oracle for whether a step is valid. That oracle supports a counterfactual training signal: the judge's score should change when the bridge entity is wrong and should not change when only surface form changes. A judge adapted this way should show three properties. First, its step score should respond far less to graded style perturbations, measured as a "dose-response" slope over continuously increasing cue intensity. Second, it should keep or improve its ability to separate valid from invalid steps. Third, and the decisive question, its reduced sensitivity should extend to bias types withheld from training, such as provenance and recency cues drawn from the Silent Judge protocol.

If the third property fails, debiasing is cue-by-cue whack-a-mole, and the proposal reports that as a negative result. The contribution is the training signal and the dose-response evaluation, not the choice of adapter. Soft prompt, low-rank adapter (LoRA) and steering vector are compared head to head as interchangeable ways of applying it.

Relation to Prior Work

Three works motivate the problem. The Silent Judge diagnoses hidden shortcut use but offers no fix. Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases shows that even reasoning models favor longer answers, and it mitigates this with PlanJudge, a discrete prompt that makes the judge write a rubric before scoring. Local Coherence or Global Validity? shows that reinforcement training makes reasoning traces locally coherent (up to about 85.6–96%) without making them globally valid. That is exactly the case where a fluency-loving judge fails. We differ from all three by training the judge against counterfactual pairs rather than prompting or diagnosing it.

Two works shape the step-level setting. AgentEval judges agent traces step by step with GPT-4o and finds that 63% of failures propagate from upstream steps. DPRM learns step rewards for multi-hop QA implicitly from final outcomes, using knowledge graphs. Neither controls for style confounds. DPRM's outcome-derived step labels can also inherit the very shortcuts we target, because a fluent wrong step attached to a lucky final answer still gets credit. Our labels come from the per-hop gold answers instead.

The method draws on three sources. P-Tuning v2 supplies deep prompt tuning: learned vectors inserted at every layer, matching full fine-tuning while training 0.1–3% of parameters. We reuse it on a judge instead of a task model. RankSteer and Causal-Steer supply the competing approach, activation steering. In activation steering, a direction is extracted from hidden states and added with a scalar weight $\alpha$ at inference. RankSteer orthogonalizes evidence directions against decision directions for single-passage ranking, and Causal-Steer isolates continuous style directions. We borrow their idea of a continuous scalar knob as an evaluation lens: we sweep cue intensity like a slider. We also treat "subtract a style direction" as a strong zero-training baseline.

The Rogue Scalpel and the representation engineering survey constrain the design. They report that steering can degrade safety and reasoning, and that pushing $\alpha$ too far causes generation to collapse into repetitive or nonsensical text. We therefore measure whether the judge keeps its general abilities under every intervention.

Research Program

Building the counterfactual step corpus. From MuSiQue training questions with gold decompositions, we generate reasoning chains with an open model and segment them into hop steps. Each step is labeled valid if it states the gold single-hop answer and that answer appears in the gold supporting paragraph.

We then apply two families of edits.

  • Substance edits. These replace the bridge entity with a plausible distractor taken from retrieved non-gold paragraphs, keeping wording fixed.
  • Style edits. These are applied at graded doses $d \in \{0,1,\dots,D\}$: added restatement sentences, confident hedging-free tone, fake citation markers, and an "EXPERT" or "LLM" provenance tag. A separate rewriting model applies them and is checked to preserve the bridge entity.

Provenance and recency cues are held out from training entirely and reserved for transfer tests.

Training objective. Let $s_P(x)$ be the frozen judge's log-odds of answering "valid" for step $x$ given the question and the evidence so far, with only the deep prompt $P$ trainable. Let $x^{+}$ and $x^{-}$ be a valid step and its substance-edited counterpart, and let $T_d$ be a style edit at dose $d$. We minimize

$$\mathcal{L}(P) = \mathbb{E}\big[\max(0,\, m - s_P(x^{+}) + s_P(x^{-}))\big] + \lambda\, \mathbb{E}\big[(s_P(x) - s_P(T_d(x)))^2\big],$$

where $m$ is a margin and $\lambda$ trades sensitivity against invariance. The style term is applied to valid and invalid steps alike, so the judge cannot satisfy it by simply scoring everything low.

Judges. We use frozen open-weight models named in the file: Qwen2.5-32B-Instruct, plus Llama3-8B as a smaller judge. Soft-prompt training needs gradients through the judge, so closed API judges are not used.

Baselines

Unmodified judge. The naive reference point is the frozen judge with a plain step-grading instruction. All gains are reported against it.

PlanJudge. The strongest existing method is PlanJudge from Reasoning Model Is Superior LLM-Judge. It is the only published mitigation in the file aimed squarely at superficial judge bias, and it raised DeepSeek-R1's BiasBench overall score from 65.00 to 97.50. We run it on the same judges, same steps and same evidence. Its rubric-generation step gets the same number of prompt-wording trials as our hyperparameter search. Beating it means a lower dose-response slope on held-out cue types with equal or better valid/invalid AUROC on MuSiQue steps. Because PlanJudge may already be near ceiling on BiasBench, we report BiasBench but do not make it the decisive metric. For step-level judging on MuSiQue counterfactuals, no reported result was found on Lacuna for any method, so all numbers there will be produced fresh.

Activation steering. This is the continuous-control competitor and needs no training. Following Causal-Steer, we extract a style direction as the geometric median of hidden-state differences between $T_d(x)$ and $x$. Following RankSteer, we orthogonalize it against a substance direction from $x^{+}$ versus $x^{-}$. At inference we subtract the style direction with weight $\alpha$. Layer and $\alpha$ are grid-searched on the same validation split our prompt uses.

LoRA on the same objective. A LoRA adapter trained on the identical corpus and loss, with matched parameter count and training steps, separates the effect of the training signal from the effect of the adapter type.

Ablations. Two ablations isolate the key ingredient. One removes the invariance term, setting $\lambda = 0$, which reduces the method to ordinary step-label tuning. The other replaces per-hop gold labels with outcome-derived labels, in the spirit of DPRM.

Score correction. A post-hoc baseline regresses the judge's score on response length and subtracts the fitted effect.

Why not just…?

Why not just use PlanJudge? It is cheap and strong on length bias. However, The Silent Judge shows that cue effects operate without ever surfacing in the judge's written reasoning, and a rubric written in text may not touch a cue the judge never verbalizes. That is an argument, not evidence. The held-out provenance and recency transfer test settles it: if PlanJudge flattens those dose-response curves as well as the tuned prompt does, the training adds nothing and we will say so.

Why not just fine-tune with LoRA? This is a real threat, and we expect LoRA trained on our corpus to perform comparably. That is why the claim rests on the counterfactual multi-hop signal rather than on soft prompts. The remaining argument for the prompt is practical: it is a detachable file of under 3% of parameters (P-Tuning v2) that leaves the shared judge untouched. We test whether that matters by measuring RewardBench accuracy retention for both adapters. DeepSeek-R1 scores 91.18% there per Huang et al., which gives a sense of the ceiling.

Why not just subtract a steering vector? It needs only a handful of examples; RankSteer used 5 anchor queries. If style bias lives on one linear direction, steering should match our method. If style cues are spread across several directions, or if steering damages judging ability as The Rogue Scalpel warns, the learned prompt should win. The $\alpha$ sweep shows which case holds, including the point where the judge degrades.

Why not just correct for length post hoc? That fixes one measurable cue and nothing that cannot be counted, such as tone or provenance. We include it only as a floor.

Proposed Experiments

The first and decisive check measures dose-response slopes on held-out MuSiQue questions. The slope is the change in judge score per unit of style dose, computed separately for trained cue types and for withheld provenance and recency cues. We also compute Verdict Shift Rate exactly as in The Silent Judge, on its ELI5 and LitBench pairs. This tests transfer to new cues and to tasks outside multi-hop QA; the GPT-4o reference there is +30% for recency on ELI5. Success means withheld-cue slopes fall substantially relative to the unmodified judge and PlanJudge, while substance AUROC does not drop.

The second check is cross-dataset step judging. We apply the same corpus construction to 2WikiMultiHopQA and HotpotQA. On Bamboogle, which lacks decompositions in the form we need, we use manual hop labels on a small sample. This tests whether the judge learned MuSiQue's wording or something more general.

The third check is downstream use. The judge scores candidate steps in an IRCoT-style retrieve-and-reason loop (IRCoT) with beam search, as Process Reward Agents does in medicine. We report MuSiQue answer F1 and support F1, with reference points of L-RAG's 0.352 accuracy on MuSiQue (L-RAG) and human performance of 78.0 answer F1 (MuSiQue). Success means the debiased judge yields higher answer F1 than both the unmodified judge and PlanJudge under equal beam width. It should also yield fewer correct-answer chains containing an invalid intermediate step, the error type that AgentEval found end-to-end checking misses (failure detection recall 0.41 versus 0.89 for step-level tracking).

The proposal is falsified in two ways. If invariance appears only for trained cue types, the counterfactual signal does not generalize. If the steering-vector baseline matches the tuned prompt on every measure, a single linear direction suffices and training is unnecessary.