TL;DR
AutoRefine is a framework that automatically calibrates LLM-based code judges using "free" but noisy signals from developer workflows, such as whether a code change was merged or abandoned. By employing an "informed teacher" LLM to filter out unexplainable outcomes, it significantly boosts judge accuracy and identifies when a judge lacks the necessary context to make a correct decision.
What Problem Does This Paper Address?
In modern software engineering, LLMs are increasingly used as "judges" to evaluate code changes (diffs) that lack automated tests—think refactors, performance tweaks, or documentation updates. However, these judges are only as good as their prompts, which must be carefully calibrated to match the specific task and the evolving nature of the codebase.
The standard way to calibrate a judge is against a manually labeled dataset, which is expensive, slow, and quickly goes "stale" as the underlying AI agents or code distributions change. While developer workflows provide a constant stream of outcome labels (e.g., "Did the tests pass?" or "Was this PR merged?"), these labels are notoriously noisy. A PR might be abandoned because the developer went on vacation or the project's scope changed, not because the code was bad. AutoRefine aims to harness these noisy workflow outcomes to keep code judges accurate without constant human intervention.
Methods
The core of AutoRefine is an iterative optimization loop that treats the judge's prompt as a set of parameters to be tuned.
- The Informed Teacher: Before calibration begins, a "teacher" LLM reviews the training data. Crucially, the teacher sees both the code change and the final outcome. Its job is to justify the label based only on the code. If the outcome cannot be explained by the code (e.g., a "FAIL" label for code that looks perfect), the teacher abstains (SKIPs the sample).
- The Calibration Loop: The deployable judge (which only sees the code, not the outcome) runs on the teacher-curated samples.
- Disagreement Analysis: When the judge and teacher disagree, an LLM analyzes the reasoning of both and proposes "balanced" prompt edits—ensuring that for every new reason to fail a patch, a corresponding reason to pass one is added to prevent the judge from becoming overly pessimistic.
- Prompt Rewriting: The prompt is updated, and the process repeats until a target agreement level is reached or a maximum number of iterations is hit.
Datasets
The researchers evaluated AutoRefine using two primary sources from an industrial environment:
- Execution Oracle Dataset: 417 records of code changes with definitive "pass/fail" test results. This served as a "ground truth" benchmark to validate the method.
- Performance-Optimization Agent Dataset: A set of 512 diffs generated by an AI agent designed to optimize code. Labels here were "landed" (merged) vs. "abandoned."
Key Insights and Ideas
- Abstention as a Signal: The teacher’s ability to "skip" samples is a powerful denoising tool. If the teacher abstains on a high percentage of samples, it’s a signal to the engineering team that the judge doesn't need a better prompt, it needs better inputs (more context).
- Noise Reduction: In a controlled test where 25% of labels were intentionally flipped to introduce noise, the teacher set aside flipped "FAIL" labels at twice the rate of genuine ones, effectively cleaning the dataset.
- The "Stale Judge" Problem: The study confirmed that a judge calibrated for an older version of an agent performs poorly on a newer version. This highlights that calibration isn't a one-time task but a continuous lifecycle requirement.
- Human-Teacher Alignment: When engineers were asked to review diffs where the teacher had abstained, they also struggled to identify failures, confirming that the teacher correctly identifies "invisible" issues that require more than just a code-level view.
Key Results
AutoRefine demonstrated significant improvements across different judge types and datasets:
- Oracle Benchmark: Calibration raised the chance-corrected agreement (Scott’s $\pi$) of a rubric-based judge from 0.36 to 0.60.
- Failure Recall: For a pass/fail judge, calibration nearly doubled the failure recall (from 0.38 to 0.73), meaning it caught significantly more buggy patches that the default prompt missed.
- Real-World Recalibration: On the performance-optimization agent, recalibrating the judge raised its agreement with merge/abandon outcomes from $\pi$ 0.16 to 0.51 on current code changes.
Limitations and Future Work
- Model Family Bias: The study used LLMs from the same family (Claude) for the teacher and the judge, which might introduce a "self-preference" bias.
- Sample Size: The calibration and test sets were relatively small (hundreds of diffs), which is common in industrial settings but limits statistical power.
- Invisible Failures: While the teacher is good at filtering out noise, it can still be "tricked" by failures that look like correct code (e.g., a logic error that appears plausible).
- Future Directions: The authors suggest that incorporating reviewer comments as "side information" for the teacher could further sharpen its justifications and the resulting judge prompts.
Summary updated