Large language models are now routinely used as automated graders, an approach called "LLM-as-a-judge." One practical problem is that the same answer, written in a different language, often gets a different grade. Across 25 languages, judges agree with themselves only weakly on parallel content, with average Fleiss' kappa around 0.3 (How Reliable is Multilingual LLM-as-a-Judge?). This happens even though the models' internal representations of different languages converge in the middle layers (MEXA), and even though they appear to contain shared, language-independent reasoning units (The Emergence of Abstract Thought in Large Language Models Beyond Any Language). The logic seems to be shared inside the model. The verdict is not.
We propose to train a step-by-step reasoning grader on machine-generated deductive proofs. Each proof is rendered in parallel in many languages, and every step carries an exact validity label. The grader is trained so that its verdict on a step is correct and identical across all renderings of that step. The central question is whether a grader taught this invariance on synthetic logic, where ground truth is free and world knowledge plays no role, becomes a more consistent and accurate judge on real multilingual reasoning it never saw: math solutions, open instruction responses, and languages held out of training. Because the same data supports a controlled comparison of English-only versus multilingual supervision, the study also tests an open contradiction in the literature about which of the two works better for evaluators.
Core Claim
Existing multilingual judges and reward models learn from data that is multilingual but unpaired. Nothing in their training directly penalizes giving different verdicts to the same content in two languages. We claim that pairing is the missing ingredient. The claim is that a process judge (a grader that scores each intermediate reasoning step rather than only the final answer) trained with an explicit cross-language agreement term on parallel synthetic proofs will do two things:
- It will cut inter-language inconsistency on held-out natural benchmarks.
- It will match or beat the same model trained on the same data without pairing.
Formal logic is the right training ground for three reasons. Step validity is mechanically checkable. Premises can be counterfactual, so the judge cannot fall back on world knowledge. Models are known to be weak at it: GPT-4 reaches 0.9% proof accuracy on depth-8 deductions in JFLD.
This is distinct from the closest work in three ways:
- Multilingual process reward modeling. Demystifying Multilingual Reasoning in Process Reward Modeling trains step scorers on translated math data mixed across languages. It stays within math, where numerals and equations anchor the logic across languages. It never ties parallel instances together.
- Cross-lingual auto evaluation. Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs (HERCULE) grades whole responses against English references. It does not assess steps and does not enforce consistency.
- Cross-lingual chain-of-thought tuning. XCOT aligns reasoning across languages through code-switched training. It does so for solvers, not graders.
Research Program
Building the multilingual proof corpus. Following the construction behind JFLD, a generator samples proof trees over abstract propositions. The trees use rules such as modus ponens and conjunction elimination, at depths 1 to 8. Native speakers then write surface templates that realize each rule, predicate and connective in English, Japanese, Hindi, Turkish, Swahili, Bengali and Telugu. Two further languages, Thai and Hungarian, are kept entirely out of training.
Corrupted proofs are made by applying one controlled error at a chosen step. The error types are:
- a wrong rule application;
- a dropped or doubled negation;
- a swapped quantifier;
- a premise cited that does not appear in the context;
- a fluent but unsupported "hallucinated" premise.
To probe the style bias documented by LLM-as-a-Judge & Reward Model: What They Can and Cannot Do, some invalid steps are deliberately written more verbosely than the valid ones. Because every rendering comes from the same tree, parallelism is exact rather than dependent on machine translation. Template quality is still a real cost, and native-speaker validation of a sample in each language is part of the plan.
Training the judge. The judge is a generative model based on Llama-3.1-8B, the same base as HERCULE, with a 70B replication if compute allows. For each step it writes a short rationale and then a verdict $v \in \{\text{valid}, \text{invalid}\}$. Let $x_s^{(\ell)}$ be step $s$ of a proof rendered in language $\ell$, $y_s$ its true label, and $p_\theta(v \mid x_s^{(\ell)})$ the judge's verdict probability. Training minimizes
$$\mathcal{L} = \sum_{\ell} \mathrm{CE}\big(p_\theta(\cdot \mid x_s^{(\ell)}), y_s\big) + \lambda \sum_{\ell \neq \ell'} \mathrm{JS}\big(p_\theta(\cdot \mid x_s^{(\ell)}),\, p_\theta(\cdot \mid x_s^{(\ell')})\big).$$
Here CE is cross-entropy, JS is Jensen–Shannon divergence between the verdict distributions for two renderings, and $\lambda$ sets the weight of cross-language agreement. Setting $\lambda = 0$ gives exactly the unpaired multilingual mixture, which is the key ablation.
Resolving the English-versus-multilingual contradiction. Two prior results point in opposite directions:
- Cross-lingual Transfer of Reward Models in Multilingual Alignment finds that English-only reward models beat target-language ones. It attributes this to representation collapse under non-English fine-tuning.
- The multilingual process reward modeling paper finds English-only step scorers inferior to multilingual mixtures.
These papers differ in at least two respects at once: outcome versus step supervision, and a scalar classification head versus a generative rationale. On our corpus we cross those two factors with English-only versus multilingual data, holding everything else fixed. The aim is to determine which factor drives the reversal. We also repeat Hong et al.'s singular-value analysis of hidden states to check whether the agreement term prevents representation collapse.
Mechanistic check. For each language we compute MEXA-style middle-layer alignment, which measures how closely a language's internal representations match English's. We then test whether that language's residual verdict inconsistency tracks this alignment score before and after training. If it does, this would begin to explain why internal alignment has so far not produced consistent judgments.
Baselines
Unaided judge. The naive reference point is the same base model prompted to grade each step in the step's own language, with a rubric. Alongside it we run GPT-4o in the same setup as the strongest available off-the-shelf judge; Son et al. report it among the top performers. All prompted systems share one rubric, and each gets the same small budget for prompt tuning on an English development split.
Unpaired multilingual step judge. This is the strongest existing method for the core question. It is the multilingual process reward modeling recipe, reimplemented on our corpus with identical data, steps and compute. The only change from our method is $\lambda = 0$, and we train it in two forms: once with a scalar head, as in the original, and once generatively. On MGSM and MATH500 that paper reports its multilingual mixture beating English-only training by up to 1.2 points and single-language training by up to 1.5 points. No reported result was found on Lacuna for step-level judging of natural-language deductive proofs. Beating this baseline means the paired judge shows higher cross-language agreement on the held-out natural benchmarks, and does so without losing step accuracy.
English-only judge. This is the Hong et al. setting and the English-only arm of the process reward modeling paper. It is trained on the English renderings only, with an equal step budget.
English-reference evaluation. On RECON we compare against HERCULE's English-reference evaluation, which reports average Cohen's kappa of 0.73 against humans, versus 0.64 for GPT-4o. Here the goal is to match HERCULE while improving consistency across languages, not necessarily to surpass it.
Symbolic ceiling. We parse each synthetic step with the generator's own grammar and check it with a prover. This is an in-domain upper bound, not a competitor.
Why not just…?
Why not just translate everything to English and use an English judge? Do Multilingual Language Models Think Better in English? shows that translation quality becomes the bottleneck in low-resource languages. Negations and quantifiers are exactly what translation tends to mangle, and they are exactly what our corruptions target. Even so, translate-test may well be competitive in high-resource languages. We run it with NLLB-3.3B and with self-translation, and report results per language and per corruption type. If it matches our judge everywhere, the paired training is not worth its cost.
Why not just prompt the judge to reason in English? Reasoning in English helps frontier-scale solvers (Language models are multilingual chain-of-thought reasoners). However, that paper also shows the effect fails at smaller scale. Separately, Fu & Liu found that prompting with explanations and rubrics left judge consistency near 0.3. We include an English-rationale prompt arm at both 8B and GPT-4o scale. We also include a trained variant whose rationales are always in English, so that the pivot language is not confounded with training.
Why not just autoformalize the steps and run a prover? On our synthetic corpus this would work almost perfectly, which is why it serves as the ceiling. Its limitation is that it needs a parser for the text being judged. Natural math solutions and open responses have no reliable parser, and those are the target regime. The proposal's value therefore rests entirely on transfer to the natural benchmarks. If the gains stay confined to synthetic logic, we will report the idea as having failed.
Why not just mix languages without pairing, as multilingual process reward models already do? This is the most serious threat, and the $\lambda = 0$ ablation answers it directly. If the unpaired mixture closes the consistency gap equally well, then multilingual exposure alone is sufficient, and the agreement term adds nothing.
Proposed Experiments
In-domain evaluation. On held-out proof trees we report three measures. The first is step-level F1 for flagging invalid steps. The second is accuracy at locating the first invalid step. The third is cross-rendering consistency, measured as Fleiss' kappa across the parallel versions of each step. Every measure is broken down by depth, corruption type, verbosity condition, and seen versus held-out language.
Transfer evaluation. We test three things without further training:
- inter-language judgment consistency on the XQuAD, WikiLingua and MGSM portions of Fu & Liu's suite, where GPT-4o reaches roughly 0.54 on binary judgments and 0.32 on 1–5 grades;
- agreement with human labels on RECON;
- best-of-N answer selection on MGSM and translated MATH500, using the protocol of the multilingual process reward modeling paper.
Decisive checks. Three outcomes determine whether the idea holds:
- The paired judge beats the unpaired judge on consistency, and on the held-out languages Thai and Hungarian, in the natural transfer sets, with step accuracy not lower.
- The verbose-invalid condition shows a smaller acceptance rate for fluent errors than the base judge.
- The factorial experiment identifies whether step supervision or generative rationales explain the English-versus-multilingual reversal.
A null result on the first check, with success only in-domain, would indicate that language-invariance learned on logic templates does not carry over to real judging.
References
- How Reliable is Multilingual LLM-as-a-Judge?
- MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language
- JFLD: A Japanese Benchmark for Deductive Reasoning Based on Formal Logic
- Demystifying Multilingual Reasoning in Process Reward Modeling
- Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
- XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning
- LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
- Cross-lingual Transfer of Reward Models in Multilingual Alignment
- Do Multilingual Language Models Think Better in English?
- Language models are multilingual chain-of-thought reasoners