Teams now grade most model outputs with a large language model acting as a judge, and correct its mistakes using a small set of human labels. The standard correction comes from prediction-powered inference, a family of estimators that average the cheap judge scores over everything and then subtract the judge's average error measured on the human-labeled subset. This gives valid confidence intervals, but only if the labeled subset comes from the same distribution as the data being evaluated. In a live evaluation stream that assumption fails in a specific way. Human labels arrive days or weeks late, so the correction is always estimated on older traffic. Meanwhile the prompts, the candidate models, and the judge's error pattern keep changing.
This note proposes to make that staleness the object of study. The key observation is simple. If the judge's score is first converted into a calibrated probability, then the prediction-powered correction term is exactly the judge's calibration-in-the-large error, meaning the gap between its average predicted probability and the average human outcome. Using an old correction is therefore biased by exactly how much the judge's calibration has drifted since the labels were collected. That turns a vague worry, "PPI assumes no shift," into a quantity that can be estimated. We use it to choose, at each evaluation period, how far back to reach for human labels. The choice follows an adaptive-window rule borrowed from drift-robust conformal prediction. The resulting confidence intervals account for label delay and widen only as much as the measured drift requires.
Core Claim
Under chronological evaluation with delayed human labels, a prediction-powered estimator built on a recalibrated judge, with its correction window chosen by a Lepski-type bias-variance rule, will keep near-nominal coverage of each period's true human win rate. It will do so where three alternatives fail. A static tuned estimator undercovers. Using only fresh labels keeps coverage but gives intervals several times wider. The classical sensitivity/specificity correction stays valid only when the judge's class-conditional error rates are stable over time. The claim is falsifiable: if static PPI++ already covers on chronological splits of Chatbot Arena, the problem this note targets does not exist at practical scale.
Background and Gap
The statistical machinery comes from Prediction-Powered Inference and its tuned variant PPI++. PPI averages model predictions over a large unlabeled set and adds a "rectifier," the mean prediction error on a labeled set. PPI++ shrinks that rectifier with a weight $\lambda$ so variance never exceeds the labeled-only estimate.
Efficient Inference for Noisy LLM-as-a-Judge Evaluation is the closest prior work and the one this proposal most directly extends. It derives the efficient influence function for judge-based win rates and shows tuned PPI++ is asymptotically efficient for binary outcomes. It reports intervals 35% to 55% narrower than standard PPI and 3x to 15x narrower than the Rogan-Gladen estimator on arena-human-preference-140k. The same paper states that these guarantees rest on the calibration and test sets sharing a distribution, and it evaluates only static cross-sectional splits. We keep its estimator as the efficiency target and drop the same-distribution assumption in the temporal direction.
The competing tradition is measurement-error correction. How to Correctly Report LLM-as-a-Judge Evaluations applies the Rogan-Gladen estimator: it estimates the judge's sensitivity and specificity on human-labeled data and inverts them. It shows the estimate stays unbiased when the true accuracy shifts, provided those class-conditional error rates are fixed. That paper constrains the design. It identifies one kind of drift, label shift, under which a stale correction is harmless, so any proposal must beat it there on width rather than validity. It also never tests whether sensitivity and specificity are stable across time, which this proposal measures directly.
Three further lines motivate the problem.
Anytime-valid, Bayes-assisted, Prediction-Powered Inference gives confidence sequences that allow continuous monitoring. It still assumes a fixed data-generating distribution, so it handles repeated looks but not drift.
On the Impact of Industrial Delays when Mitigating Distribution Drifts shows that ignoring labeling and deployment delays overstates drift mitigation by up to 39.86%. This is why our protocol never lets an estimator see labels younger than the delay $d$.
Systematic Review of Approaches to Preserve ML Performance under Temporal Dataset Shift in Clinical Medicine finds that calibration deteriorated over time in 11 of 15 studies while ranking held up. That is exactly the failure that biases a stale rectifier.
The method for choosing a window comes from Distribution-Free Predictive Inference under Unknown Temporal Drift. It picks a rolling-window size online using a Goldenshluger-Lepski bias proxy and proves an oracle inequality for conformal coverage. We transplant that rule from prediction sets to the PPI rectifier. The recalibration step follows Forecasting Model Success at Inference Time, whose isotonic recalibration cut expected calibration error from 0.12 to 0.03. That paper names online recalibration under drift as its main open problem.
The Estimator
Notation is as follows.
- Let $s(X)$ be the raw judge output for item $X$, for example its preference verdict, its verbalized confidence, or the agreement rate across position-swapped runs.
- Let $g_t$ be an isotonic map, fit on human-labeled items older than $t-d$, that turns $s$ into a calibrated probability that the human prefers response A.
- Let $\theta_t$ be the human win rate among items in period $t$.
For a window of $w$ labeled periods ending at the delay boundary, the estimate is
$$\hat\theta_t(w) = \frac{1}{N_t}\sum_{i \in \text{period } t} g_t(X_i) + \hat\lambda\,\frac{1}{n_w}\sum_{j \in [t-d-w,\,t-d]} \big(Y_j - g_t(X_j)\big).$$
Here $N_t$ is the number of judge-scored items in period $t$, $n_w$ is the number of human-labeled items in the window, $Y_j$ is the human vote, and $\hat\lambda$ is the PPI++ weight.
The bias of this estimator is exactly the change in the calibrated judge's average error between the window and now:
$$\text{bias}_t(w) = \lambda\big(\Delta_{[t-d-w,\,t-d]} - \Delta_t\big), \qquad \Delta_\tau = \mathbb{E}_\tau[Y - g_t(X)].$$
$\Delta_t$ itself is unobservable, because current labels have not arrived. Following the adaptive-window rule, we pick $w$ to minimize a computable upper bound on bias plus a sampling term. The bias part is estimated from how much the rectifier computed on nested shorter windows disagrees with longer ones. The sampling term is a standard-error term from the delta method. This bound is computed from past windows only. It therefore assumes drift is roughly smooth, so that recent change in $\Delta$ predicts change over the last $d$ periods. That assumption is stated, not hidden, and the experiments test it.
The reported interval adds the estimated bias bound to the usual PPI++ half-width. If a few fresh labels exist in the current period, they enter as a separate unbiased rectifier and are combined by inverse variance.
Calibration matters here for two reasons beyond tidiness. When drift is purely in which prompts appear, so that the human-given-judge relationship is fixed, a calibrated $g$ makes $\Delta_\tau$ near zero in every period and long windows are safe. A raw judge score has a rectifier that moves with the prompt mix even when nothing about the judge changed.
Research Program
The work has three parts.
The first part is empirical and comes before any method. On Chatbot Arena human votes ordered by timestamp, we measure three drifting quantities per period, using all human labels as ground truth. These are the calibrated judge's rectifier $\Delta_\tau$, the judge's sensitivity and specificity, and the raw-score rectifier. This tells us directly which invariance, if any, holds in practice. For judges we use a GPT-4-class model under the de-biased pairwise protocol of Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, which runs both orders to remove position bias, plus a cheaper judge such as the Gemini-3-Flash judge used by the Rogan-Gladen paper.
The second part is the estimator above. It comes with a finite-sample coverage statement of the oracle-inequality type: coverage at least nominal minus a term that vanishes when drift per period is bounded, and width within a constant of the best fixed window in hindsight.
The third part is a stress test. We inject controlled drift into the stream: we swap the judge model mid-stream, change the mix of candidate model pairs, and vary the label delay $d$ from zero to several periods. This maps where each estimator breaks.
Baselines
The naive reference point is the raw judge win rate with a binomial interval. How to Correctly Report LLM-as-a-Judge Evaluations reports that raw scores deviate from human win rates by 5 to 10 percentage points and that naive intervals reach 0% coverage in simulation. We expect the same here and include it only to anchor scale.
The validity-only reference is the classical mean of whatever human labels exist in the current period. It is unbiased under any drift but wide. Any proposed method must be narrower than it at matched coverage, or it has no reason to exist.
The strongest existing method is static tuned PPI++ / EIF as in Efficient Inference for Noisy LLM-as-a-Judge Evaluation, with the rectifier fit on all labeled data older than the delay boundary. Its reported advantage on arena-human-preference-140k is intervals 35% to 55% narrower than standard PPI and 3x to 15x narrower than Rogan-Gladen. No reported result was found on Lacuna for its coverage under chronological splits. We run it with the same judges, the same calibration labels, the same delay, and the same $\hat\lambda$ estimation. Beating it means holding coverage within two points of 95% across periods where it falls below, while staying within 20% of its width in periods where it covers.
The transport baseline is Rogan-Gladen with sensitivity and specificity from past labels. It is the method designed for shifting accuracy, and its reported performance is restored 95% coverage and 37% exact ranking recovery with Gemini-3-Flash, as against 0% for raw scores. Beating it means matching its coverage while giving narrower intervals in periods where its error rates are stable, and higher coverage where they are not.
Two further comparisons isolate the proposal's ingredients. A fixed-window PPI++ swept over window lengths gives the best fixed window chosen in hindsight. This is an oracle that tests whether adaptive selection adds anything. An ablation that runs our adaptive estimator on the raw, uncalibrated judge score isolates the contribution of recalibration.
All methods share labels, judges, delay and evaluation periods. None may tune hyperparameters on future periods.
Why not just…?
Why not just use static PPI++ and accept a little bias? If the calibrated rectifier barely moves over the Arena timeline, this is the right answer, and the first experiment is designed to find out. The reasons to expect otherwise are indirect. Calibration degraded over time in 11 of 15 clinical studies in the temporal shift review, and Arena's candidate pool turns over as models are released. Still, there is no direct evidence on Lacuna for judges. If coverage of static PPI++ stays near nominal, we report that as the finding and drop the method.
Why not just use Rogan-Gladen, which already transports across accuracy shift? It is valid exactly when sensitivity and specificity are fixed. Under label shift alone it is a real threat on validity, and our method must win there on width, which the efficiency comparison suggests is plausible given its 3x to 15x wider intervals. Rogan-Gladen also ignores covariates such as length and topic, which are known to sway judges. Our drift measurements will show directly whether its error rates are stable over time. Where they are, we will recommend it.
Why not just recalibrate the judge on the most recent labels and report the mean calibrated score? That is our estimator with the rectifier set to zero. It is biased whenever the map has drifted during the label delay, and it offers no interval that reflects this. The ablation with $\lambda=0$ measures exactly what the rectifier and window rule add.
Why not just pick a sensible fixed window, say the last month? This is the most serious competitor. A well-chosen fixed window may perform almost as well as adaptive selection. The hindsight-best fixed window is included for that reason. If it matches our method across judges and delays, the honest conclusion is that window choice is not the bottleneck. The contribution would then reduce to the recalibrated-rectifier diagnosis and the delay-aware protocol.
Proposed Experiments
The main benchmark is the Chatbot Arena human-preference data (Chatbot Arena; arena-human-preference-140k). We split it by timestamp into weekly periods and treat each period's full human vote as ground truth for $\theta_t$. If the release lacks fine timestamps, we order by the release dates of the candidate models, which still yields a directed sequence of shifting candidate pools.
The labeled budget per period is varied from 50 to 500 items, and the delay $d$ from 0 to 4 periods. Metrics are the following:
- per-period and worst-decile coverage of 95% intervals;
- mean interval width;
- mean squared error of $\hat\theta_t$;
- Kendall's $\tau$ between estimated and true model rankings per period;
- the per-period expected calibration error of $g_t$, reported alongside so that coverage loss can be tied to calibration drift.
AlpacaEval and MT-Bench serve only as static controls. On them, all PPI variants should cover, so any loss there points to a bug rather than to drift.
The decisive checks are four.
- Static PPI++ should cover nominally on a random split of Arena but undercover on the chronological split. Without this gap there is nothing to fix.
- The rectifier of the calibrated judge should drift measurably less across periods than the raw-score rectifier.
- Under the injected judge-swap, the adaptive window should shrink within a few periods and coverage should recover. Fixed windows should not recover.
- Coverage of the adaptive method should degrade gracefully, not collapse, as delay $d$ grows. This tests the smooth-drift assumption the bias proxy relies on.
References
- Prediction-powered inference
- PPI++: Efficient Prediction-Powered Inference
- Efficient Inference for Noisy LLM-as-a-Judge Evaluation
- How to Correctly Report LLM-as-a-Judge Evaluations
- Anytime-valid, Bayes-assisted, Prediction-Powered Inference
- On the Impact of Industrial Delays when Mitigating Distribution Drifts: An Empirical Study on Real-World Financial Systems
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine
- Distribution-Free Predictive Inference under Unknown Temporal Drift
- Forecasting Model Success at Inference Time: Calibrated Probabilistic Forecasts for Cost-Optimal LLM Cascades
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference