AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes

Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin Håkanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan NagappanarXiv 2026 · arXiv

TL;DR

AutoRefine is a framework that automatically calibrates LLM-based code judges using "free" but noisy signals from developer workflows, such as whether a code change was merged or abandoned. By employing an "informed teacher" LLM to filter out unexplainable outcomes, it significantly boosts judge accuracy and identifies when a judge lacks the necessary context to make a correct decision.

What Problem Does This Paper Address?

In modern software engineering, LLMs are increasingly used as "judges" to evaluate code changes (diffs) that lack automated tests—think refactors, performance tweaks, or documentation updates. However, these judges are only as good as their prompts, which must be carefully calibrated to match the specific task and the evolving nature of the codebase.

The standard way to calibrate a judge is against a manually labeled dataset, which is expensive, slow, and quickly goes "stale" as the underlying AI agents or code distributions change. While developer workflows provide a constant stream of outcome labels (e.g., "Did the tests pass?" or "Was this PR merged?"), these labels are notoriously noisy. A PR might be abandoned because the developer went on vacation or the project's scope changed, not because the code was bad. AutoRefine aims to harness these noisy workflow outcomes to keep code judges accurate without constant human intervention.

Methods

The core of AutoRefine is an iterative optimization loop that treats the judge's prompt as a set of parameters to be tuned.

Fig. 1. The AutoRefine loop (orange: LLM calls; gray: data). The informed teacher curates the training labels once: it keeps a label with a justification grounded in the judge’s view, or abstains (SKIP) and drops the sample. Each iteration runs the deployable judge on the kept samples and scores it against their labels; disagreement analysis contrasts the judge’s reasoning with the teacher’s justification and proposes balanced prompt edits. An uncurated validation split selects the final prompt.
  1. The Informed Teacher: Before calibration begins, a "teacher" LLM reviews the training data. Crucially, the teacher sees both the code change and the final outcome. Its job is to justify the label based only on the code. If the outcome cannot be explained by the code (e.g., a "FAIL" label for code that looks perfect), the teacher abstains (SKIPs the sample).
  2. The Calibration Loop: The deployable judge (which only sees the code, not the outcome) runs on the teacher-curated samples.
  3. Disagreement Analysis: When the judge and teacher disagree, an LLM analyzes the reasoning of both and proposes "balanced" prompt edits—ensuring that for every new reason to fail a patch, a corresponding reason to pass one is added to prevent the judge from becoming overly pessimistic.
  4. Prompt Rewriting: The prompt is updated, and the process repeats until a target agreement level is reached or a maximum number of iterations is hit.

Datasets

The researchers evaluated AutoRefine using two primary sources from an industrial environment:

  • Execution Oracle Dataset: 417 records of code changes with definitive "pass/fail" test results. This served as a "ground truth" benchmark to validate the method.
  • Performance-Optimization Agent Dataset: A set of 512 diffs generated by an AI agent designed to optimize code. Labels here were "landed" (merged) vs. "abandoned."
Breakdown of the diffs used in the performance-optimization agent study.

Key Insights and Ideas

  • Abstention as a Signal: The teacher’s ability to "skip" samples is a powerful denoising tool. If the teacher abstains on a high percentage of samples, it’s a signal to the engineering team that the judge doesn't need a better prompt, it needs better inputs (more context).
  • Noise Reduction: In a controlled test where 25% of labels were intentionally flipped to introduce noise, the teacher set aside flipped "FAIL" labels at twice the rate of genuine ones, effectively cleaning the dataset.
  • The "Stale Judge" Problem: The study confirmed that a judge calibrated for an older version of an agent performs poorly on a newer version. This highlights that calibration isn't a one-time task but a continuous lifecycle requirement.
  • Human-Teacher Alignment: When engineers were asked to review diffs where the teacher had abstained, they also struggled to identify failures, confirming that the teacher correctly identifies "invisible" issues that require more than just a code-level view.

Key Results

AutoRefine demonstrated significant improvements across different judge types and datasets:

  • Oracle Benchmark: Calibration raised the chance-corrected agreement (Scott’s $\pi$) of a rubric-based judge from 0.36 to 0.60.
  • Failure Recall: For a pass/fail judge, calibration nearly doubled the failure recall (from 0.38 to 0.73), meaning it caught significantly more buggy patches that the default prompt missed.
Quantitative results showing the improvement in judge performance after calibration.
  • Real-World Recalibration: On the performance-optimization agent, recalibrating the judge raised its agreement with merge/abandon outcomes from $\pi$ 0.16 to 0.51 on current code changes.
Comparison of judge performance before (v1) and after (v2) recalibration across different code slices.

Limitations and Future Work

  • Model Family Bias: The study used LLMs from the same family (Claude) for the teacher and the judge, which might introduce a "self-preference" bias.
  • Sample Size: The calibration and test sets were relatively small (hundreds of diffs), which is common in industrial settings but limits statistical power.
  • Invisible Failures: While the teacher is good at filtering out noise, it can still be "tricked" by failures that look like correct code (e.g., a logic error that appears plausible).
  • Future Directions: The authors suggest that incorporating reviewer comments as "side information" for the teacher could further sharpen its justifications and the resulting judge prompts.

Summary updated

Abstract

LLM judges increasingly serve as quality gates for code changes that have no test oracle, but a judge prompt must be calibrated for its task and recalibrated as the system it judges evolves, while hand labels for the calibration are resource intensive and go stale. Development workflows emit outcome labels for free (tests pass or fail, changes land or are abandoned), but these labels are noisy. We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels, with an informed teacher that sees the judge's input plus the outcome, explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. On a held-out dataset with an execution oracle, calibration raised chance-corrected agreement (Scott's $π$) from 0.36 to 0.60. On a deployed agent's outcome labels, with 25\% flipped at random to introduce noise, the teacher set aside flipped \textsc{fail} labels at more than twice the rate of genuine ones, cutting overall label noise from 25\% to 20\%. Recalibrating that agent's judge using AutoRefine raised its agreement with merge/abandon outcomes on 17 independently held-out diffs from the current workflow ($π$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves. By abstaining, the teacher not only removes noisy labels but also flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.
BibTeX
@misc{shahidi2026autorefine,
  author = {Shervin Shahidi and James Poon and Hang Xie and Kenneth O. Xu and Robin Håkanson and Vikram Nair and Jens Pillgram-Larsen and Nachiappan Nagappan},
  title = {{AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes}},
  year = {2026},
  eprint = {2611.09291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09291},
}
Figure