Is a Code Judge Only as Good as Its Weakest Language? A Parallel-Program Audit

Large language models are now routinely asked to judge code: to read a program and say whether it is correct, without running it. These judgments grade benchmarks, rank candidate solutions, and supply rewards for training. Every careful study of how reliable these judges are has used a single language, usually Python, or a single formal language such as Lean 4. Code models are known to be much weaker in less common languages such as Haskell, OCaml, or Fortran. The untested worry is that a judge trusted on Python will be quietly unreliable on those languages. It may fall back on superficial cues like comments, names, and familiar-looking structure exactly where it understands the code least.

This proposal builds parallel programs to measure that directly. These are the same task with the same injected bug, written in eleven languages and checked by execution. Because the content is held fixed, any change in judge accuracy can be attributed to the language rather than to problem difficulty. Our central hypothesis has two parts. First, a model's reliability as a judge in a language tracks its ability to write code in that language. Second, susceptibility to superficial cues grows as that ability shrinks. We then test whether two popular fixes close the gap. One is translating the code into Python before judging. The other is the judge simulating execution step by step. We also test whether a judge's agreement with itself across translations can warn practitioners that it is unreliable in a new language when no labels are available.

Core Claim

Prior per-language comparisons mix two things: the language, and the fact that different languages come with different problems. Once the problems are held fixed, we predict that judge error on logic-altering bugs rises as the judge's generation accuracy in that language falls. We also predict that the accuracy lost to superficial perturbations, such as a # correct code comment or a misleading task description, is larger in low-resource and functional languages than in Python. If both hold, single-language audits of code judges systematically overstate their reliability. If neither holds, and judgment transfers across languages even where generation does not, that is also a useful finding. It would mean judging relies on different competence than writing, and the parallel design is what lets us tell these outcomes apart.

Relation to Prior Work

Three studies motivate the claim by showing that code judges are fragile, and all three stay within one language. What can Large Language Models Capture about Code Functional Equivalence? builds SeqCoBench, a set of Python program pairs that differ by edits either preserving meaning (renaming, dead code) or altering it (operator inversion, flag flips). It finds that models miss single-token logic flips and correlate only weakly with execution (r ≈ 0.31). Don't Judge Code by Its Cover shows judges lose up to 26.7 percentage points of accuracy under superficial perturbations on single-language AtCoder code. Certifying the Judge tests Lean 4 judges against "metamorphic" properties, meaning rules a good judge must obey, such as unchanged scores under variable renaming. It finds that frontier models break the renaming rule. We reuse the edit categories from the first paper and the perturbation taxonomy from the second. From the third we borrow the idea of treating judges as software to be tested against properties. What we add is the language axis, with content held fixed.

Two cross-language works constrain the design. McEval shows generation collapses on functional and procedural languages and attributes much of this to failures in following instructions rather than missing knowledge. That raises the possibility that judging, which needs less instruction-following, degrades less. Our experiment is built to detect exactly that possibility. McEval also supplies reference solutions and tests across 40 languages. Agnostics turns tasks into standard input/output problems run in generic containers. This makes execution-based ground truth affordable for languages like OCaml and Fortran, and it is the tool we use to verify every item.

MCTS-Judge is both the strongest method and a point of tension. It has the judge search over self-written tests and simulate execution line by line, and it reports raising DeepSeek-Coder-V2-16B from 41.0% to 80.0% accuracy on APPS. It includes HumanEval-X but reports no per-language analysis, and the file lists no separate HumanEval-X number. SeqCoBench's weak execution correlation, and the Python-only finding of A Differential Fuzzing-Based Evaluation of Functional Equivalence that about 21.65% of broken refactorings pass all benchmark tests, both suggest caution about trusting simulated execution. We address this directly by checking simulated execution traces against real ones in each language.

Research Program

Building the parallel item set. We start from tasks that have reference solutions in many languages. HumanEval-X covers Python, C++, Java, JavaScript and Go, as introduced with CodeGeeX. McEval adds Rust, Haskell, Scheme, OCaml, Julia and Fortran, giving eleven languages that span static and dynamic typing and object-oriented, procedural and functional styles. For each task, and each semantics-altering edit class from SeqCoBench (operator inversion, boundary shift, boolean flip, comparison reversal), we locate the aligned position in every language's solution and apply the language-appropriate form of the same edit. The alignment is proposed by a model and spot-checked by a person.

An item enters the set only if execution in Agnostics-style containers confirms that the edited program fails and the unedited one passes. We check this on the benchmark tests plus extra randomly generated inputs, because thin test suites can hide broken programs. Meaning-preserving edits (renaming, statement reordering, dead code) are applied and verified the same way. Onto every item we layer the cover perturbations from the bias study: authority comments, self-declared correctness, misleading task text, and illusory complexity.

The result is a grid of task × edit × perturbation × language in which only the language varies along one axis. Language-specific bug types that have no parallel, such as C integer overflow or Rust ownership misuse, are collected as a separate, clearly labeled supplement and never pooled into the matched analysis.

Measurement. Each judge sees the task description and one candidate and returns a correct/incorrect verdict with a confidence score. We use five judges: GPT-4o, Claude 3.5 Sonnet, LLaMA-3.1-70B, DeepSeek-Coder-V2-16B and Qwen2.5-Coder-14B. The per-language quantities are:

  • accuracy on unperturbed items;
  • average precision for separating meaning-preserving from meaning-altering pairs, matching SeqCoBench's metric;
  • bias susceptibility, defined as the accuracy drop under each cover perturbation relative to the same item unperturbed.

We relate these to the same model's execution-verified generation accuracy on those languages in McEval. A mixed-effects regression with task as a random effect gives the language effect net of task difficulty.

Checking simulated execution. For MCTS-Judge we instrument the real programs to log intermediate variable values on the inputs the judge itself generated. We then score trace fidelity: the fraction of simulated intermediate states that match the real ones. This asks whether the method's gains come from faithful simulation or from confident reasoning that only looks like tracing, and whether fidelity collapses in languages with unfamiliar evaluation order or memory rules.

A label-free warning signal. Borrowing the metamorphic logic of Certifying the Judge, we define a translation-invariance property: a judge should give the same verdict to parallel versions of the same item. We measure how often each judge's verdicts agree across languages. We then test, leaving one language out at a time, whether this agreement rate predicts the judge's true accuracy in the held-out language. If it does, a practitioner could estimate a judge's reliability in, say, Fortran using only parallel programs, without writing Fortran tests.

Baselines

The naive reference point is a judge-free surface score. Following SeqCoBench, we compute the ChrF character-overlap score between a candidate and the reference solution in the same language and threshold it. That paper reports ChrF at 55.25 average precision on Python SeqCoBench, against 70.85 for CodeLlama-13B. We rerun it on our items in every language, so per-language judge results can be read against a floor that uses no understanding at all. A majority-class verdict is reported alongside.

The standard practice baseline is single-shot chain-of-thought prompting of each judge. The same prompt template is translated only in its language mentions, and the same decoding settings are used everywhere. This is what practitioners actually deploy, and it is the setting in which the bias study found drops of up to 26.7 points.

The strongest existing method is MCTS-Judge, run with its released search procedure on DeepSeek-Coder-V2-16B and Qwen2.5-Coder-14B. We use a fixed rollout budget identical across languages and cap the per-item token budget. The single-shot baseline may use self-consistency sampling up to the same token count, so compute is matched. Its published APPS figure is 80.0% accuracy with DeepSeek-Coder-V2-16B, and no per-language HumanEval-X number is reported on Lacuna. We treat the method as closing the language gap only if its accuracy spread across the eleven languages is at most half the single-shot spread, and only if trace fidelity stays high in the languages where it succeeds.

The pivot baseline translates each non-Python candidate into Python with the judge model, then judges the translation. It gets the same total token budget. Crucially, we also execute the translated program to measure the bug-preservation rate: how often a buggy original stays buggy after translation.

The key ablation removes the parallel design. We compute per-language accuracy on ordinary unmatched McEval items and compare the implied ranking of languages with the matched ranking. This shows how much of the "language effect" in naive studies is really task difficulty.

Direct execution against the verified tests is an upper reference, not a competitor. It defines the labels.

Why not just…?

Why not just execute the code, since containers now make this cheap across languages? For grading benchmarks with good tests, execution should win, and we use it as ground truth. The case for studying judges is that they are deployed exactly where execution is missing or unreliable. Examples include review of polyglot code, where CrossPL needed purpose-built state machines and C-compilation sandboxes, and tasks without test suites. The fuzzing study also shows that ordinary tests let about 21.65% of broken refactorings through. This proposal does not argue that judges beat execution. It measures how much trust judges deserve in each language, and that question is open no matter how cheap execution becomes.

Why not just translate everything into Python and judge there? This is the most serious alternative, and it may work. Our concern is that translation models tend to write the intended program, silently repairing the injected bug, which would turn buggy candidates into correct-looking ones. The pivot baseline measures this directly through the bug-preservation rate. If preservation is high and pivot accuracy matches Python-native accuracy in every language, the practical answer is "pivot." Our contribution would then narrow to documenting the gap and certifying when pivoting is safe, and we would report it that way.

Why not just compare judge accuracy per language on existing multilingual benchmarks? Different languages in McEval come with different problems and differently hard solutions, so a lower Haskell score could reflect harder Haskell tasks rather than a weaker Haskell judge. The unmatched-versus-matched ablation quantifies this confound. If the two rankings agree closely, the parallel construction was unnecessary, and we will say so.

Why not just fine-tune the judge on multilingual data, as McEval-Instruct did for generation? This is a reasonable fix, but it presupposes knowing where and how the judge fails. As a secondary arm, we fine-tune Qwen2.5-Coder-14B on execution-labeled verdicts from six languages and test it on the held-out five. That shows whether judging competence transfers across languages the way McEval found instruction-following to transfer.

Decisive Checks

The first hypothesis, that judge reliability tracks generation ability, is supported if two conditions both hold. The matched language effect on accuracy for meaning-altering edits must remain significant after the task random effect. It must also correlate with McEval generation accuracy across languages, with Spearman above 0.6 pooled over judges. The second hypothesis is supported if bias susceptibility in the functional and low-resource group exceeds Python's by a margin larger than twice its bootstrap interval. The claim is falsified if matched accuracy is flat across languages while generation varies widely.

For MCTS-Judge, a pattern in which accuracy holds in a language but trace fidelity falls below half would show that its gains there do not come from faithful simulation. The warning signal is useful if cross-language agreement predicts held-out-language accuracy with Spearman above 0.7 across judge–language combinations. Each outcome is informative in either direction, and every label comes from execution, so no result depends on another model's opinion.