LLM judges are large language models asked to grade or compare other models' outputs. They now stand in for human raters across much of model training and evaluation. They stop being reliable once the text is not English. Give a judge the same content in 25 languages and its verdicts agree only weakly: average inter-language Fleiss' kappa is around 0.3 (How Reliable is Multilingual LLM-as-a-Judge?). Judges also flip preferences, picking the correct answer in English and the wrong one for the same content in another language (M-RewardBench). The two usual fixes each break something. Translating everything into English first erases the phenomena a judge must check, such as grammar, the script used, and whether the reply is in the requested language. Translation also feeds the judge machine-translated text, which judges are known to over-reward. Asking the judge to reason natively in the target language uses the model's weaker non-English reasoning.
This note proposes a third option and tests it carefully. We train the judge to read the target-language input directly, write its reasoning in English, and quote the target-language spans its verdict depends on. The training signal costs almost nothing. Human-translated parallel passages give us an English twin for each target-language item. The same model's English verdict on that twin, kept only when it matches the gold label, becomes the target for its verdict on the target-language version. The model is therefore taught to judge other languages as well as it already judges English, with no proprietary teacher and no human preference labels in the target languages. We train on about 20 languages and measure whether consistency carries over to the long tail of the 122 languages in MM-Eval's consistency subset.
Core Claim
Much of the multilingual judging gap comes from where the judge reasons, not from what it understands. Two findings support this. Multilingual models share concept-level units across languages (Qualifying Knowledge and Knowledge Sharing in Multilingual Models). Yet they also route chain-of-thought reasoning through language-specific neurons in middle and deep layers: switching off Japanese-specific neurons cuts Japanese MGSM accuracy by 27.4% while barely affecting other languages (Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting). Taken together, these suggest a judge reasoning in Hindi relies on thinner Hindi-specific circuitry, even though its comprehension of the Hindi passage may be fine.
We claim two things. First, a judge trained to reason in English while quoting native spans will reach higher cross-language consistency and accuracy than either native-language reasoning or translate-then-judge. Second, the gain will be largest on exactly the subsets where translation should fail: linguistic correctness and language hallucination.
Relation to Prior Work
Prior work plays three roles here. Three diagnostic papers motivate the problem. Fu and Liu measure cross-language agreement of five judges on parallel XQuAD, MGSM, WMT23, WikiLingua and XDailyDialog. They find that multilingual instruction tuning (Aya-Expanse), larger scale and chain-of-thought prompting do not fix it, and that majority voting helps only partially. MM-Eval builds preference triplets in 18 languages plus a Language Consistency Index (LCI) over 122 Belebele languages; it reports open judges below 0.6 LCI against about 0.8 for proprietary ones. M-RewardBench documents losses of up to 13% relative to English, and shows generative judges are more robust than classifier reward models. All three diagnose the gap; none trains a fix. We use them as our test bed.
Two training papers are our direct competitors. M-Prometheus fine-tunes Qwen2.5 judges on rubric feedback synthesized natively in the target language by Claude-3.5-Sonnet. It finds that this beats translated English judge data, and that adding translation-evaluation data helps. Its rationales are written in the target language and come from a proprietary teacher. Ours are language-mixed and come from the model's own English mode. DIBJudge attacks a specific failure, the judge's preference for machine-translated text, using a dual-channel information-bottleneck model. That changes the architecture and adds overhead. We change only what the judge writes and what it is trained to match. Its finding also constrains our design: any training text built by machine translation risks teaching the judge to reward translationese. For that reason our parallel inputs are human-translated and our candidates are generated natively, never machine-translated.
Two papers supply the method. Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought shows that reasoning traces with English logic and Korean entities and constraints beat both all-Korean and all-English reasoning for problem solvers, avoiding the topic drift of translate-and-reason. It never tests the judge role and covers only Korean. Flex-Judge shows that about 1,000 text reasoning samples can transfer judging across modalities. That motivates our bet that a modest set of training languages will transfer to unseen ones. Finally, Ebing and Glavaš show that translation-based transfer beats zero-shot transfer for low-resource languages, by 18.9% on AmericasNLI. Translate-then-judge is therefore a serious competitor, not a strawman.
Research Program
The training data are built from FLORES-200 dev passages. These are professionally translated, so each target-language passage has a true English twin. We exclude every passage that appears in Belebele, because Belebele is built on FLORES text and drives MM-Eval's consistency subset.
For each passage we create judging items in four formats: pointwise yes/no correctness, pointwise 1–5 grades, pairwise preference, and a language-fidelity check. Questions are posed natively in each language. Candidate answers are produced natively in that language by several open generators of different quality, so they are never machine-translated. Where an answer can be checked against the passage, the item carries a gold label.
To get training targets, we run the base judge in two ways. On the English twin it reasons and judges normally. On the target-language item it is prompted to produce a language-mixed rationale: English reasoning that quotes the decisive target-language spans verbatim, followed by a verdict. We keep a sampled rationale only if three conditions hold: its verdict matches the gold label, it matches the English-twin verdict, and every quoted span exists in the input.
The judge is then fine-tuned on these filtered rationales, which is a form of rejection-sampling self-distillation. For the 1–5 grade format we add a second stage using regression-aware reinforcement learning from REAL (code). The reward is the negative absolute distance to the gold grade, plus a smaller term for distance to the English-twin grade. This uses the ordinal structure that binary rewards discard, aimed at the format where Fu and Liu found agreement weakest.
We train on about 20 languages and hold out everything else. Backbones are Qwen2.5-7B-Instruct and Qwen3-8B, which match the bases of M-Prometheus and DIBJudge.
Alongside the training work we run a mechanistic check. Using the same language-specific-neuron method as the cross-lingual chain-of-thought paper, we test whether trained judges depend less on target-language-specific neurons than native-reasoning judges. We measure this by how much switching those neurons off degrades judging accuracy.
Baselines
The naive reference is the untrained backbone prompted in the target language with the standard rubric and native chain-of-thought. This is the setting Fu and Liu measured at kappa of about 0.3 across models. Every trained variant must clearly beat it on both accuracy and consistency.
The strongest external reference is a frontier judge prompted directly. On M-RewardBench, GPT-4 Turbo reports 83.5% average accuracy across 23 languages and GPT-4o reports 81.1% (M-RewardBench). Proprietary judges reach LCI of about 0.8 on MM-Eval, against an open-model ceiling of about 0.6 (MM-Eval). GPT-4o's peak cross-language kappa is 0.54 for yes/no and 0.32 for grades. Reaching the proprietary LCI of about 0.8 with an 8B open model would count as a clear win. So would exceeding the 0.32 grade kappa on Fu and Liu's parallel sets.
The strongest open competitors are M-Prometheus-14B, which is reported to surpass GPT-4o on MM-Eval, and DIBJudge-Qwen3-8B, reported as best among open models on M-RewardBench. For neither did we find a reported numeric score on Lacuna. We will re-evaluate both released models on identical test items and prompts. We will also train an M-Prometheus-style native-rationale judge on our exact items and backbone, with the same number of examples and gradient steps. This last comparison isolates rationale language from data source.
The translate-then-judge baseline translates the target input with NLLB, judges it in English with the same backbone, and adds a frontier-judge variant. Its fine-tuned form follows Ebing and Glavaš's roundtrip recipe, which trains on translated and back-translated data so training noise matches test noise. The majority-vote ensemble of open judges, which Fu and Liu found helpful, is run at matched inference cost.
Ablations isolate the key ingredients. We compare language-mixed, all-English and all-native rationales under identical filtering and training budget. We remove the English-twin agreement filter, keeping gold-only filtering. We remove the gold filter, keeping twin agreement only, which tests whether consistency is bought by copying English mistakes. Finally, we drop the REAL stage.
Why not just…?
Why not just translate into English and use a strong English judge? This is the most serious alternative, and it may well win on reasoning-heavy subsets. It cannot win where the thing being judged lives in the source language. Translation erases grammatical errors and wrong-language replies, and DIBJudge shows translated text distorts what judges prefer. M-RewardBench also finds that better translation quality moves judge accuracy by only 1–3%, so translation noise alone does not explain the gap. We will report results subset by subset. If translate-then-judge matches us on MM-Eval's Linguistics and Language Hallucination subsets, our central claim fails and we will say so.
Why not just prompt the judge to reason in English and quote spans, with no training? This could capture much of the gain for large models, and it is cheap. We run it as a baseline on every backbone. If prompting alone closes most of the gap at 7–8B, the contribution shrinks to a prompting result, and we will report it that way. Our expectation that training matters rests on a specific observation: Aya-Expanse's multilingual tuning did not improve consistency, which suggests small models do not keep a reasoning mode they were never trained to keep.
Why not just train on native target-language rationales, as M-Prometheus does? That recipe works, but it needs a proprietary teacher and covers about 20 languages. The matched native-rationale ablation answers whether rationale language matters once data are held fixed. The held-out-language LCI answers whether either approach transfers beyond its training languages.
Why not just ensemble several judges? Fu and Liu found voting improves consistency, but it multiplies inference cost and leaves each member's bias in place. We compare at equal inference compute.
Proposed Experiments and Decisive Checks
Evaluation covers MM-Eval's five core subsets in 18 languages and its 122-language consistency subset (LCI), M-RewardBench in 23 languages, and Fu and Liu's five parallel corpora. On the parallel corpora we report Fleiss' kappa for both yes/no and grade criteria, and we hold out the languages used for training. Results are reported separately by script family, Latin versus Indic, Perso-Arabic and others, because Found in Translation finds consistency collapses on non-Latin scripts, down to 0.11 for Aya-Expanse on Indic.
Four checks decide the outcome. First, LCI on the roughly 100 consistency-subset languages never seen in training must rise relative to native reasoning; otherwise the transfer claim fails. Second, accuracy against gold must not fall while consistency rises. A judge that gives every language the same wrong score is consistent and useless, so we report both together for every model. Third, on the Linguistics and Language Hallucination subsets, language-mixed judging must beat translate-then-judge. Fourth, quote fidelity: in trained rationales, the share of quoted spans that actually exist in the input should stay near 100%, and when quoted spans are corrupted the verdict should change. That would show the judge really grounds its verdict in the native text.
As a translationese probe, we follow DIBJudge's Belebele protocol and compare preference for human versus machine-translated text before and after training. If our training increases that preference, the self-distillation is leaking English-shaped judgments, and we will add DIBJudge's bias channel as a follow-up rather than claim the problem solved.
References
- How Reliable is Multilingual LLM-as-a-Judge?
- M-RewardBench: Evaluating Reward Models in Multilingual Settings
- Qualifying Knowledge and Knowledge Sharing in Multilingual Models
- Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting
- MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
- M-Prometheus: A Suite of Open Multilingual LLM Judges
- Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
- Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
- To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages
- REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
- Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate