Do Step-Level Judges Reward Translationese in Multilingual Multi-Hop Reasoning?

Language models increasingly answer multi-hop questions, which need several linked facts ("who directed this film, and where was that person born?"), by writing out their reasoning one step at a time and retrieving a document for each step. A second model, used as an automatic grader and usually called an LLM judge, then scores each intermediate step. Those step scores decide which reasoning path a search procedure keeps, which trajectories get reused as training data, and which failures developers see. Separately, researchers have shown that judges prefer text that reads like a machine translation from English ("translationese") over natural, native-written text.

Our question is where these two findings meet. When the reasoning is written in German, Spanish or Greek, does the step judge accept a wrong step more easily if it is phrased in translationese than if it is phrased idiomatically? If so, any search or self-training loop driven by that judge will quietly select for English-shaped reasoning in other languages, whether or not that reasoning is correct.

We propose a controlled measurement and a targeted fix. The measurement is a factorial benchmark of multi-hop reasoning steps. Each step exists in four versions: correct or incorrect, crossed with native or translationese phrasing. The evidence the step relies on is held fixed across all four. This lets us separate how much of a judge's verdict comes from what a step claims and how much comes from how it sounds. The fix is a judge that first rewrites each step into short, standalone factual claims and checks each claim against the retrieved passage before giving a verdict. We compare it directly against the obvious cheaper remedy of translating everything to English before judging.

Core Claim

We hypothesize that translationese and the judge's tendency to believe the reasoning it reads are not independent effects. Instead they compound, so that fabricated or wrong steps are accepted at a higher rate when phrased in translationese. We also hypothesize that the effect is largest in high-resource languages, where models themselves produce the most translationese. If this holds, it has a concrete downstream consequence. Judge-guided tree search, which explores several reasoning paths and keeps the ones a judge scores highest, will pick trajectories that are more translationese-like and no more accurate. Evaluation built on the same judge will then overstate multilingual reasoning quality.

What Prior Work Establishes and Leaves Open

Four papers motivate the hypothesis.

  • Translationese bias in judges. Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck (DIBJudge) shows that judges prefer machine-translated over native text. It traces the bias to English-aligned internal representations and to token probabilities, and it reduces bias severity by 80% on BELEBELE with a dual-encoder information-bottleneck judge. It studies only single-turn reading comprehension, open-ended generation, summarization and reward benchmarks. It never looks at intermediate reasoning steps.
  • Belief in the agent's own reasoning. Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation shows that judges believe an agent's self-reported progress. Fabricated progress statements raise false positive rates by 20–30 absolute points on frontier judges. That study is English-only, so it cannot say whether phrasing style modulates this gullibility.
  • Cross-language reliability. How Reliable is Multilingual LLM-as-a-Judge? reports average Fleiss' kappa around 0.3 for judges across 25 parallel languages. This is consistent with language-dependent noise, but the study stays single-turn.
  • Translationese in model outputs. An Investigation of Translationese in the Generations of Multilingual Large Language Models trains classifiers on part-of-speech patterns and function-word statistics. Its German classifier flags 60.2% of Gemini's and 37.6% of Llama's "native" generations as translationese, and 0% of human native text. So the reasoning steps a judge sees in these languages will often be translationese to begin with.

Four more papers supply the setting and part of the method.

None of these works crosses language register with step correctness. None asks whether a process judge's selections shift the register of the trajectories it keeps.

Two papers constrain the design.

Research Program

Building the benchmark. We draw questions from MuSiQue (♫ MuSiQue: Multihop Questions via Single-hop Question Composition), chosen because it is built so that single-hop shortcuts fail. For each question we run an IRCoT-style pipeline to obtain 2–4-hop traces, each step paired with the passage retrieved for it. The target languages are German, Spanish and Greek, where validated translationese classifiers exist. We add Italian from the XHOPREASON language set, with human naturalness ratings standing in for a classifier.

For each step we create two versions.

  • Native version. A native speaker writes the step idiomatically from the English meaning.
  • Translationese version. The step is machine-translated from English, then round-tripped through English once more to strengthen artifacts without changing content.

We cross these two versions with a correctness manipulation. There are two error types: a wrong bridge entity (the intermediate entity that links one hop to the next), and progress fabrication in the style of Gaming the Judge, where the step claims the passage says something it does not. The fabricated content is fixed first, then rendered in both styles. The supporting passage is identical across all four cells of each step.

We run two manipulation checks. First, the classifier from the translationese investigation paper must separate the two versions. Second, bilingual annotators must confirm that the paired versions are equivalent in meaning. Pairs that fail either check are dropped. The target is roughly 400 traces per language, about 1,200 steps and 4,800 judged items per language. Native rewriting is the main cost, which is why the language set stays small.

Measuring bias. For every judge we compute two quantities.

  • Register effect: the difference in acceptance rate between translationese and native versions, holding correctness fixed. We estimate it separately for correct and incorrect steps.
  • Interaction: whether the register effect is larger on incorrect steps.

We fit a mixed-effects logistic model with item and language as random effects. The "compounding" hypothesis corresponds to a positive interaction between register and incorrectness. We also report error-detection area under the ROC curve (AUROC) separately for each register.

The fix: a claim-grounded judge. The judge first breaks each step into short, standalone claims in the step's own language. It then checks each claim against the retrieved passage, and only then aggregates into a step verdict. The idea is that breaking a step into atomic claims strips away most of its surface style, and checking against evidence removes the need to trust the author's assertions. These two ingredients address the two biases separately, and we ablate them separately.

Baselines

Outcome-only judging (naive reference). Grade only the final answer against the gold answer, or ask the judge only whether the final answer is right. It is immune to step phrasing but blind to where a chain broke. We include it so that step-level judging has to justify itself. It uses the same traces and the same judge model. Its register effect should be near zero by construction, and its step-error detection is at chance by construction.

Step judge with a rubric and chain-of-thought (strongest existing method). A judge that reasons before its verdict and is given upstream context and the retrieved passage, following AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking. Reasoning before the verdict follows TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge. It is the strongest competitor because it is what current step-level pipelines actually deploy. Its reported English performance is failure-detection recall 0.89 with Cohen's kappa 0.84. No reported result was found on Lacuna for step-level judging of multilingual multi-hop reasoning traces. We run it with the same judge backbones (GPT-4o, Qwen-2.5-72B, Llama-3.3-70B) and the same passages. The rubric prompt gets the same budget of 20 prompt-development iterations on a held-out English split as our method does.

DIBJudge. The representation-level debiasing judge from Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck. Its reported result is an 80% average reduction in bias severity on BELEBELE. We run it if the Qwen3-8B weights are available. Otherwise we retrain its disentanglement objective on the same backbone using the paper's recipe. It tests whether a debiasing method trained on single-turn texts transfers to reasoning steps.

Translate-to-English judging. Every step and passage is translated to English before a rubric step judge scores it.

Ablations. Two ablations isolate the key ingredient. Claims without evidence tests whether style stripping alone removes the register effect. Evidence without claim decomposition is a plain step judge given the passage and told to verify it; it tests whether grounding alone suffices.

What counts as beating the baselines. The claim-grounded judge beats the strongest baseline if two conditions hold together. First, its 95% interval for the translationese-versus-native acceptance gap on incorrect steps includes zero while the rubric judge's does not. Second, its step-error AUROC on native text is at least as high as the best baseline's.

Why not just…?

Why not just translate everything to English before judging? This is the most serious threat, and it may simply work. Translation into English maps both the native and the translationese version toward similar English sentences, which could erase the register effect for free. The case against it rests on two risks.

These are hypotheses, not evidence. The experiment settles the question by reporting this baseline's register effect and its native-text AUROC next to ours. If it closes the gap without losing accuracy, that becomes the recommended practice and our judge is unnecessary.

Why not just ask the judge to reason more carefully? Reasoning before judging improves agreement with humans, but Gaming the Judge shows that reasoning judges still accept fabricated progress at sharply higher rates. Their reasoning takes the agent's assertions at face value instead of checking them. The rubric chain-of-thought baseline tests this directly on multilingual steps.

Why not just use DIBJudge? It is trained to detach judgments from signals of text origin on single-turn texts. A reasoning step's correctness depends on a passage and on earlier steps, which its training never saw. Transfer is an open empirical question that the DIBJudge baseline answers directly. If DIBJudge closes the gap, the finding is still new, because it would show that representation-level debiasing carries over to process judging.

Why not just use self-consistency voting and skip step judges? Majority voting over sampled answers avoids judges altogether. It gives no step-level signal, though, and GRAT-style search and self-training need a per-node score. Voting is included only as an accuracy reference in the downstream search test.

Proposed Experiments and Decisive Checks

Experiment 1: does the compounding effect exist? The first result is the register-by-correctness interaction for each judge and language on the factorial benchmark. The finding fails if the interaction is null, with a tight interval around zero, across all judges. We would report that negative result, since it would suggest that English truth-bias findings transfer without a register component. A secondary prediction follows from the translationese investigation paper's observation that high-resource languages carry more translationese: the register effect should be larger in German and Spanish than in Greek.

Experiment 2: does the bias change what search selects? The downstream check runs GRAT-style tree search on MuSiQue questions posed in each target language, once with the rubric step judge and once with the claim-grounded judge as the process scorer. We measure final-answer F1, and we measure the translationese-classifier rate of the selected trajectories against the rate over all expanded trajectories. The key prediction concerns selection bias. With the rubric judge, the selected paths should be more translationese than the pool at equal accuracy. With the claim-grounded judge, this gap should shrink. If self-training on the selected trajectories for one round raises the translationese rate of the model's fresh generations, that establishes the feedback loop. As an anchor for the search setup, GRAT reports 75.2% accuracy on 2WikiMultiHopQA with Llama3-70B in English.

Experiment 3: are the verdicts trustworthy? A human check on 300 steps per language has bilingual annotators label step correctness blind to condition. We report judge-human Cohen's kappa for each register, against AgentEval's English 0.84 as the reference.