Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation

Masoud FetanatarXiv 2026 · arXiv

TL;DR

LLM-based judges for code evaluation can be dangerously insensitive to catastrophic model failures. This study demonstrates that even when a model starts producing syntactically invalid Python code in over 60% of its attempts, its "judge-based" scores barely decline, highlighting a critical need for simple "syntax gates" in evaluation pipelines.

What Problem Does This Paper Address?

Evaluating code generation is a bottleneck in AI research. While running code against unit tests (execution-based evaluation) is the gold standard, it is slow, requires secure sandboxing, and depends on the quality of the test suite. Consequently, many researchers have turned to "LLM-as-a-judge"—using a powerful model like GPT-4 or Qwen-72B to read and grade generated code without actually running it.

The authors tackle a fundamental question: Is a high judge-based score a reliable signal of quality? Specifically, they investigate whether these judges can "hide" substantial damage to a model's output—such as a sudden explosion in syntax errors—that would be immediately obvious to a compiler or a human developer.

Methods

The researchers conducted a "replay study," repeating an evaluation of Qwen-2.5-7B-Instruct models that had undergone Reinforcement Learning with Verifiable Rewards (RLVR) using the GRPO algorithm. They compared three versions of the model:

  1. Base: The original instruct model.
  2. CE-trained: Rewarded by execution tests.
  3. SLJ-trained: Rewarded by a strong LLM judge.

They evaluated these models using three different "scorers":

  • LLM Judge: Qwen-2.5-72B-Instruct acting as a grader.
  • Judge + Syntax Gate: A hybrid approach where code must be approved by the judge and pass a basic Python syntax check (ast.parse).
  • Execution: A "hardened" test runner that executes the code against unit tests.

The primary metric used was Pass@k, which estimates the probability that at least one out of $k$ generated samples is correct.

Datasets

The study utilized a total of 421 coding problems drawn from two standard benchmarks:

  • HumanEval: 164 problems.
  • MBPP (Mostly Basic Python Problems): 257 problems from the sanitized test split.

Key Insights and Ideas

  • Judges are "Too Kind" to Broken Code: The most startling finding is that LLM judges frequently approve code that is syntactically impossible to run. For the trained models, the judge approved roughly 60% of samples that failed a basic Python parse check.
  • Pass@k Masks Failure: Because Pass@k only requires one success out of many attempts, it can hide a high "trash" rate. If a model fails 60% of the time but produces one lucky valid answer in 32 tries, its Pass@32 score remains high.
  • The "Signal Loss" at Scale: As the number of attempts ($k$) increases, the LLM judge loses its ability to distinguish between a healthy model and a "damaged" one.
  • Specific Failure Modes: The "damage" observed wasn't random; the trained models developed a weird habit of adding extra closing brackets (e.g., for i in range(1, 10))):). While a human or a compiler sees this as "impossible code," the LLM judge often overlooked it.
Figure 1: Left: how often each model wrote code that does not parse, in the first run and in the second. The base model’s rate remains nearly the same. Both trained checkpoints jump from under 1% to over 60%. The scale is logarithmic, so the short bars are visible. Right: how far apart the base model and the damaged SLJ-trained checkpoint look, measured as a score difference in percentage points (pp) on the same second-run outputs, as the number of attempts allowed per problem grows. Running the code keeps them clearly apart. The judge does not: its gap has almost vanished by 128 attempts. Scores and gaps are listed in Appendix J.

Key Results

  • Syntax Explosion: In the replayed run, the trained checkpoints produced invalid Python in 63% to 67% of attempts, whereas the base model remained stable at <1%.
  • Insensitive Scores: Despite this massive drop in code validity, the SLJ-trained model's Pass@32 score under the strong judge fell by only ~3 percentage points (from 96.67% to 93.95%).
  • Vanishing Gaps: At 128 attempts, the judge saw almost no difference between the base model and the damaged model (a 0.71 pp gap), while execution tests showed a massive 15.91 pp difference.
Table 2: Four checks on the same second-run answers. Parsing measures the percentage of individual answers that pass Python’s syntax check. The other columns report Pass@128: the estimated chance of at least one success in 128 attempts per question. Success means judge approval, approval plus passing the syntax check (gate), or passing execution tests, respectively. Judge scores are similarly high for all three models, whereas the other checks distinguish the base from the trained models. CE-trained and SLJ-trained remain close.
  • The Power of the Syntax Gate: Adding a simple, zero-cost syntax check (the "gate") before the judge revealed the damage immediately, dropping the Pass@1 score for the damaged models by over 37 points.

Limitations and Future Work

The authors acknowledge that the exact cause of the "damage" in the replayed checkpoints remains unresolved—it could be due to checkpoint corruption, loading issues, or subtle changes in the serving environment. However, this ambiguity actually strengthens their point: evaluations must be sensitive enough to flag such changes, regardless of the cause.

Future Work & Recommendations: * Report the Parse Rate: The authors strongly recommend that every judge-based evaluation should also report the percentage of answers that pass a basic syntax check. * Sensitivity Audits: Researchers should check how sensitive their evaluation pipeline is to intentional "damage" (like adding extra brackets) to ensure the judge isn't just hallucinating correctness. * Retain Artifacts: To make these discrepancies easier to investigate, researchers should save generated answers, judge verdicts, and model checksums.

Summary updated

Abstract

Language models are often used to judge generated code without running it. A useful evaluation should reveal substantial changes in the answers it scores. We examine this requirement by repeating an earlier evaluation on 421 coding problems, comparing Qwen-2.5-7B-Instruct with two versions trained using Group Relative Policy Optimization (GRPO). One receives rewards from execution tests (CE-trained), and the other from a strong language-model judge (SLJ-trained). The base model's syntax-failure rate remains stable, but the trained checkpoints produce invalid Python on 63%-67% of attempts. Despite this change, the strong judge's Pass@32 scores fall by only about 3 percentage points, from 96.67% to 93.95% for SLJ-trained. Within the later run, allowing more attempts makes the base and trained checkpoints look increasingly similar under the judge. With 128 attempts per question, the judge accepts at least one SLJ-trained answer on 96.44% of problems, compared with 28.27% under execution tests. We recommend reporting the share of answers that pass Python's syntax check alongside every judge-based score. This check requires no tests or code execution and flags invalid answers that the judge accepts, although it cannot establish functional correctness. Our findings show how a high judge-based score can hide substantial output damage and motivate checking an evaluation's sensitivity to change. Project page and code: https://curiousorbitnotes.github.io/evorobust-replay-audit/
BibTeX
@misc{fetanat2026impossible,
  author = {Masoud Fetanat},
  title = {{Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation}},
  year = {2026},
  eprint = {2611.01572},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01572},
}
Figure