AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration

Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing LiaoarXiv 2025 · DOI · arXiv

Introduction: The Evaluation Bottleneck

Evaluating the quality of code generated by Large Language Models (LLMs) has become a critical bottleneck in modern software engineering. While existing benchmarks effectively measure whether a model can solve a specific coding puzzle, they lack the robustness required to evaluate the "LLM-as-a-Judge"—the models tasked with critiquing and scoring that code.

A new framework, AXIOM (Assessment via X-verification and Oracle-guided Mutation), addresses this gap by systematically "breaking" high-quality code to test whether AI judges can accurately detect and categorize the resulting defects.

Limitations of Current Benchmarks

Traditional metrics like BLEU or CodeBLEU rely on surface-level text similarity, which fails to capture functional correctness or architectural quality. Furthermore, existing LLM-as-a-judge benchmarks suffer from three primary deficiencies:

  1. Subjectivity: Reliance on vague Likert scales (e.g., "mostly helpful") leads to inconsistent human ground truth labels.
  2. Coarse Granularity: Binary pass/fail labels ignore the nuances of "almost correct" code that requires only minor refinement.
  3. Data Skew: Benchmarks derived from open-source repositories are often skewed toward high-quality, working code, failing to represent the "long tail" of messy, buggy code LLMs frequently produce.

The AXIOM Framework

AXIOM introduces a two-stage process designed to create a balanced and objective benchmark for code evaluation.

The two-stage AXIOM framework: Rule-Based Perturbation and Multisource Quality Calibration.

Stage 1: Rule-Guided Perturbation

Rather than asking an LLM to generate "bad code" from scratch—a process that is difficult to control—AXIOM takes a "perfect" reference program and applies a sequence of 45 specific perturbation rules to degrade it. These rules are mapped to a 0-5 "Refinement Effort" scale, providing an objective measure of how much work is required to return the code to a production-ready state:

  • Score 5: Production-ready; no effort needed.
  • Score 4/3: Functionally perfect but requires minor/major refactoring for quality.
  • Score 2/1: Functionally broken; requires minor/major repair.
  • Score 0: Fundamentally flawed or irrelevant.
Examples of perturbation rules mapped to specific score ceilings and the distribution of rules across quality levels.

Stage 2: Multisource Quality Calibration

To ensure the ground truth is reliable, AXIOM utilizes a streamlined human annotation process. Experts are provided with a "diagnosis report" containing the code diff, static analysis results (e.g., Pylint), execution reports, and an LLM-generated quality summary. The human's role is to calibrate the target score based on this multi-faceted evidence, a method that proved significantly more accurate than blind grading.

AXIOMBench: A Multilingual Benchmark

The authors introduced AXIOMBench, a large-scale benchmark comprising 1,962 code evaluation problems across Python, C++, and Java. Derived from high-difficulty sources like BigCodeBench and LiveCodeBench, AXIOMBench features a balanced distribution across all score levels (0–5). This allows for a rigorous test of a judge's ability to distinguish between subtle quality issues and major functional flaws.

Key Insights and Results

The study yielded several critical findings regarding the current state of LLM judges:

  • Human Alignment: The AXIOM calibration process achieved a 96.4% Krippendorff’s alpha, indicating near-perfect agreement among human experts on the ground truth labels.
  • Ranking vs. Absolute Scoring: While top models like DeepSeek-V3.1 and Gemini-2.5-Flash showed high Spearman correlation (up to 0.69) for ranking code, their absolute agreement with human scores was much lower (alpha between 0.45 and 0.63).
  • The "Agentic" Consistency Penalty: Surprisingly, complex "agentic" judges—those using multiple steps and tools—were less consistent than simpler judges. The stochastic nature of LLMs causes uncertainty to compound at every extra step in a workflow.
  • Defect Detection Bias: LLM judges exhibit a strong tendency toward pessimism. While they are adept at finding bugs, they frequently hallucinate flaws in correct code, leading to high false-positive rates.
  • The Refinement Gap: No tested LLM judge could reliably estimate "refinement effort." Models frequently misclassified localized fixes as massive architectural changes, resulting in a Matthews Correlation Coefficient (MCC) of less than 10.0 for that specific task.

Conclusion and Future Work

AXIOM demonstrates that while current LLMs are promising as code rankers, they struggle with objective quality scoring and refinement estimation. Future work will involve extending the framework to more languages, such as Rust and JavaScript, and exploring methods to mitigate the hallucination bias inherent in current LLM judges. By shifting the focus from abstract "quality" to concrete "refinement effort," AXIOM provides a more interpretable and objective path forward for automated code evaluation.

Summary updated

Abstract

Large language models (LLMs) have been increasingly deployed in real-world software engineering, fostering the development of code evaluation metrics to study the quality of LLM-generated code. Conventional rule-based metrics merely score programs based on their surface-level similarities with reference programs instead of analyzing functionality and code quality in depth. To address this limitation, researchers have developed LLM-as-a-judge metrics, prompting LLMs to evaluate and score code, and curated various code evaluation benchmarks to validate their effectiveness. However, these benchmarks suffer from critical limitations, hindering reliable assessments of evaluation capability: Some feature coarse-grained binary labels, which reduce rich code behavior to a single bit of information, obscuring subtle errors. Others propose fine-grained but subjective, vaguely-defined evaluation criteria, introducing unreliability in manually-annotated scores, which is the ground-truth they rely on. Furthermore, they often use uncontrolled data synthesis methods, leading to unbalanced score distributions that poorly represent real-world code generation scenarios. To curate a diverse benchmark with programs of well-balanced distributions across various quality levels and streamline the manual annotation procedure, we propose AXIOM, a novel perturbation-based framework for synthesizing code evaluation benchmarks at scale. It reframes program scores as the refinement effort needed for deployment, consisting of two stages: (1) Rule-guided perturbation, which prompts LLMs to apply sequences of predefined perturbation rules to existing high-quality programs to modify their functionality and code quality, enabling us to precisely control each program's target score to achieve balanced score distributions. (2) Multisource quality calibration, which first selects a subset of...
BibTeX
@misc{wang2025axiom,
  author = {Ruiqi Wang and Xinchen Wang and Cuiyun Gao and Chun Yong Chong and Xin Xia and Qing Liao},
  title = {{AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration}},
  year = {2025},
  eprint = {2512.20159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.20159},
}
Figure