Language models are now routinely used as judges of code changes. They decide whether a patch is correct, or which of two translations runs faster, in settings where hidden tests are unavailable or incomplete. Two facts about these judges are usually treated separately. First, their verdicts are noisy: the same item can receive different scores on different runs, so a single sample is not a reproducible measurement. Second, they cost energy, and on a GPU most of that energy is spent in the token-by-token generation phase, which runs at the same speed almost regardless of clock frequency. This proposal puts the two facts together. The real cost of judging a code change is the energy needed to reach a verdict that is stable, one that another lab would reproduce. That cost is set mostly by how many output tokens the judge must generate across all the repeated samples it needs. How the change is shown to the judge (whole files, a raw diff, or an explicitly tagged edit) plausibly controls how noisy each sample is, and therefore how many samples are needed.
The central claim is testable and, as far as the file shows, untested. Showing a judge a structured edit representation, meaning an input where the changed tokens are explicitly marked and separated from unchanged context, reduces per-item verdict variance compared with full before/after files. Under variance-adaptive resampling, this lower variance becomes fewer generated tokens and fewer measured Joules per reproducible verdict. Shorter inputs save a little energy directly. The larger effect is that cleaner inputs make the judge less noisy, and noise is what forces repeated sampling. A secondary claim follows from the hardware physics: the input representation also shifts the energy-optimal GPU clock. Long whole-file inputs make judging dominated by the compute-bound, frequency-sensitive phase that reads the prompt. Short structured edits make it dominated by the memory-bound generation phase, where lowering the clock is nearly free.
Why this gap is real
The evidence that representation matters comes from generation, not judging. Exploring the Capabilities of LLMs for Code Change Related Tasks finds that isolated code differences beat full-file snapshots across change-related tasks. Automating code review activities by large-scale pre-training marks added, deleted, and kept lines with special tokens. Program merge conflict resolution via neural transformers adds edit-type flags to token embeddings. Pull Requests as a Training Signal for Repo-Level Code Editing replaces line-numbered diffs with Search/Replace blocks that are verified by applying them and checking the result byte for byte. All of these supply candidate representations. None of them asks whether a representation makes an evaluator more consistent.
The closest existing work is SwiftTrans, described in Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation. Its DiffSelector judges the diff between two candidate translations to rank them for correctness and speed. It optimizes the runtime of the generated program (292ms versus 528ms on SwiftBench). It does not measure the judge's own variance or energy, and its pairwise sorting chain is exactly where per-comparison noise could propagate into non-reproducible rankings.
On the judging side, MCTS-Judge raises accuracy by running a tree search over reasoning paths with simulated execution. It measures cost in tokens and never in Joules, and it evaluates code against a task description, not a change. LLM-as-a-Judge on a Budget supplies the resampling method we reuse. Its ROBIN-HOOD algorithm spends repeated queries on high-variance items, halving query cost for the same worst-case error on HelpSteer2. It was never applied to code or tied to physical energy.
The energy side supplies both constraint and method. Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling shows three things on general NLP tasks:
- Generation takes 77–91% of inference time.
- Downclocking from 2842 to 180 MHz saves about 42% energy for 1–6% more latency.
- The energy-delay optimum sits at 960 MHz.
Classic results supply the physics behind this. Reducing both dynamic and leakage energy consumption for hard real-time systems and A Survey of Architectural Techniques for Near-Threshold Computing explain why slowing down stops paying once leakage and "uncore" components (memory controllers and other always-on parts of the chip) keep drawing power for the longer run. The implication is that once the clock is set sensibly, remaining energy scales with how long the GPU is busy generating tokens, and that is set by output length times number of samples. Nobody in the file has measured this for code judges.
Core Claim
For judge sample $i$ on an item, measured energy decomposes approximately as
$$E \approx \sum_{i=1}^{n} \left[ e_{\text{pre}}(L_{\text{in}}, f) + e_{\text{dec}}(f)\, L_{\text{out},i} \right],$$
where the terms are defined as follows:
- $n$ is the number of samples needed to reach a target confidence-interval width on the verdict.
- $L_{\text{in}}$ is the input length set by the edit representation.
- $L_{\text{out},i}$ is the number of generated tokens in sample $i$.
- $f$ is the GPU clock.
- $e_{\text{pre}}$ and $e_{\text{dec}}$ are the per-sample prompt-processing energy and the per-token generation energy.
The number of samples $n$ grows with the variance of the item's verdict, because the width of a mean's confidence interval shrinks as $\sigma/\sqrt{n}$. We claim that structured edit inputs reduce $\sigma$ enough that the drop in $n$ dominates the energy saving, more than the direct drop in $L_{\text{in}}$. We further claim that the frequency minimizing $E$ is higher for whole-file inputs than for structured edits.
Research Program
We build one measurement harness, run on the NVML-instrumented GPU testbed of Characterizing LLM Inference Energy-Performance Tradeoffs (NVIDIA's management library, which reports GPU power draw). The harness records prompt-processing and generation energy separately for every judge call.
Each code change is rendered into four inputs carrying identical information:
- full before/after files;
- a standard unified diff;
- a token-tagged edit in the style of Lu et al., with keep, delete, and insert markers;
- round-trip-verified Search/Replace blocks as in Clean-PR.
The judges are open-weight models named in the file, DeepSeek-Coder-V2-16B and the Qwen2.5 family, so that others can rerun them. Each item is sampled 30 times at nonzero temperature per representation, mirroring the ROBIN-HOOD protocol. This gives a per-item variance estimate and lets us replay any allocation rule offline without rerunning the GPU.
Three analyses follow. The first is a paired comparison: the same items under different representations, testing whether variance drops. The second replays ROBIN-HOOD and uniform allocation to compute measured Joules to reach a fixed worst-case estimation error. The third sweeps the clock over the device's supported range for each representation to locate the energy-minimizing frequency.
We also test whether generated-token count alone predicts measured Joules. If it does, to within a few percent across representations and clocks, energy reporting for judges can be replaced by hardware-independent token accounting. That would itself be a useful reproducibility result, because Joule measurements do not transfer across GPUs.
Baselines
The naive reference point is what most papers do. The judge sees full before/after files, takes one sample, and runs at the default clock. It anchors both accuracy and energy, and every other arm is reported relative to it on the same items and hardware.
The second baseline is temperature-zero single sampling with the same full-file input. It is the cheapest way to appear reproducible. We report its accuracy, its agreement across repeated runs, and its Joules.
The strongest accuracy competitor is MCTS-Judge, which reports 80.0% judge accuracy on APPS with DeepSeek-Coder-V2-16B, up from 41.0% for the plain model (MCTS-Judge). We rerun it with the same backbone, the same APPS items, and the same hardware, and record its Joules. Beating it means matching its accuracy within a few points at clearly lower measured energy, or exceeding it at equal energy. We do not require exceeding 80.0% outright.
For resampling, the reference is ROBIN-HOOD versus uniform allocation. On HelpSteer2 correctness with gpt-4.1-nano, it reports worst-case error of 0.300 versus 0.451 for uniform at a 50,000-query budget (LLM-as-a-Judge on a Budget). We run both allocators with an identical query budget on every representation.
For pairwise efficiency judging, the competitor is SwiftTrans's DiffSelector. The file reports end-to-end translation results (90.2% computational accuracy, 292ms average runtime on SwiftBench), but no reported result was found on Lacuna for DiffSelector's standalone judging accuracy or variance. We therefore reimplement its pairwise protocol with the same backbone.
The key ablation holds the allocation rule and the clock fixed and varies only the representation. This isolates whether structured edits reduce variance. A second ablation holds the representation fixed and varies only the clock. This isolates the frequency effect from the representation effect.
Why not just…?
Why not just run the unit tests? Where good tests exist, execution is the right ground truth, and we use it as the label on SWE-bench Verified and APPS. But functional tests miss efficiency: SwiftTrans shows correct translations differing nearly twofold in runtime. DrainCode (DrainCode) shows adversarially verbose code passing tests while raising energy by up to 49%. Judges are used precisely where tests are missing or blind. We report test execution wall time next to judge Joules, so readers can see when a judge is not worth running at all.
Why not just set temperature to zero? It removes sampling noise within a run, but does not make the verdict correct. The plain 16B judge at 41.0% on APPS is a System-1 judge, one that answers by quick surface pattern-matching. Greedy output also hides uncertainty rather than measuring it. Our temperature-zero baseline makes this concrete. If it matches the adaptive arm's accuracy and cross-run agreement at lower energy, that is a real threat to the proposal, and we would report it as the recommended protocol.
Why not just downclock the GPU to around 960 MHz? We do this in every arm, because it is an orthogonal and nearly free saving. Once decode dominates, however, frequency gives a one-time discount, and energy still scales with total generated tokens times samples. The frequency sweep tests whether the optimal clock actually differs by representation. If it does not, the secondary claim fails and the story reduces to token and sample counts.
Why not just count tokens instead of measuring Joules? This may well suffice, and we treat it as a hypothesis, not a strawman. The harness directly tests whether token counts predict Joules. We expect the prediction to break mainly for long whole-file inputs, where prompt processing is significant and sensitive to the clock.
Proposed Experiments
We use three benchmarks, each with a different source of ground truth:
- On SWE-bench Verified, we collect candidate patches from open models, including a Clean-PR-tuned Qwen2.5-Coder-32B, which the file reports at 30.6% Pass@1. The judge predicts pass or fail, and hidden tests supply labels.
- On SwiftBench, the judge picks the faster of two correct translations, and measured runtime supplies labels.
- On APPS, the setup allows direct comparison with MCTS-Judge.
We report four metrics:
- judge accuracy against labels;
- per-item verdict variance;
- worst-case estimation error at fixed budgets;
- measured GPU Joules per reproducible verdict, split into prompt processing and generation.
The first decisive check is paired: structured edits must show lower per-item variance than full files on the same items. The second is that this reduction must yield measurably fewer Joules under ROBIN-HOOD to reach the same worst-case error. The third is that a structured-edit, adaptively sampled judge must trace an accuracy-versus-Joules curve that dominates both the naive baseline and MCTS-Judge somewhere along it.
If representation changes accuracy but not variance, the central mechanism is wrong. The result would then be reported as a cheaper-input finding only.
References
- Exploring the Capabilities of LLMs for Code-Change-Related Tasks
- Automating code review activities by large-scale pre-training
- Program merge conflict resolution via neural transformers
- Pull Requests as a Training Signal for Repo-Level Code Editing
- Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation
- MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
- LLM-as-a-Judge on a Budget
- Characterizing LLM Inference Energy-Performance Tradeoffs Across Workloads and GPU Scaling
- Reducing both dynamic and leakage energy consumption for hard real-time systems
- A Survey of Architectural Techniques for Near-Threshold Computing
- DrainCode: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning