Learning-Progress Annotation for LLM Judges

When people build an automatic evaluator from a large language model (an "LLM judge" that scores or compares model outputs in place of human raters), they still need human labels to tune and check it. Those labels are expensive, so the natural move is active learning: send humans the items the judge is most unsure about. This note argues that such uncertainty-driven selection has a specific, predictable failure. Much of a judge's uncertainty comes from items on which humans themselves disagree. On a 1–5 scale, judges' 90% prediction sets for SummEval fluency and consistency cover about 4.9 of the five points (Diagnosing LLM Judge Reliability). An uncertainty sampler keeps buying labels for these items and never learns anything from them. Reinforcement learning has a name for this trap: the "noisy TV," a screen of random static that a curiosity-driven agent stares at forever because the static never becomes predictable.

Exploration research escaped that trap by rewarding learning progress, meaning how much the agent's error actually falls, instead of rewarding surprise. We propose doing the same thing for annotation. We split the unlabeled pool into slices, for example evaluation criterion crossed with topic or source system. We then spend the human-label budget on the slices where the judge's error on freshly bought labels is still going down. Slices that are already mastered stop receiving labels, and so do slices that are hopelessly ambiguous, because in both cases progress goes to zero. The judge is trained to predict the full distribution of human ratings, so that ambiguity is represented in what it outputs rather than chased.

Core Claim

The claim is narrow and testable. When an LLM judge is fitted to human ratings under a fixed label budget, a selection rule based on measured per-slice learning progress will reach a lower held-out divergence from human rating distributions than uncertainty sampling, variance-based allocation, or random selection. The gap should grow with the fraction of the pool made up of high-disagreement items. Two parts are new. First, the selection signal is an empirical derivative of loss on fresh labels, not a model-based uncertainty estimate. Second, the target is distributional, so items that are irreducibly ambiguous hit a loss floor equal to their human entropy and produce zero progress automatically. No hand-set filtering threshold is needed.

Where This Sits

The mechanism comes from exploration. VIME rewards the change in a Bayesian posterior over dynamics, a change that vanishes on pure noise. It does this through an approximate Bayesian neural network that would be impractical to place over an LLM judge. Beyond Noisy-TVs: Learning Progress Monitoring gets the same zero-on-noise property more cheaply, by rewarding the difference between a model's past error and its current error. It reports that its coverage dropped only 12.4% under injected noise, against up to 85% for baselines. We reuse LPM's core idea. We replace its learned error network with direct measurement, because in annotation every newly bought label is a free held-out test point for the current judge. A constraining result also comes from this line of work: Beyond Markovian found that LoRA ensembles of value heads failed to capture posterior uncertainty. This is one reason we avoid relying on ensemble-based epistemic estimates for LLM-sized judges.

The target comes from Distributional LLM-as-a-Judge. That work trains judges to match the full human rating distribution and lowers SNLI KL divergence from 0.60 to 0.23. It does so on a static, pre-collected dataset and names multi-annotator cost as a limitation. We supply the missing piece: a rule for which items to annotate.

The motivating evidence for the gap is threefold. The conformal diagnostics above show that uncertainty is driven by criterion and is often near-total. Enhancing LLM-as-a-Judge through Active-Sampling-based Prompt Optimization selects items by predicted-score entropy plus diversity. It reaches its MSE target labeling about 16% of SummEval training samples, against 32–50% for random and density baselines. However, it overfits in later iterations, which is consistent with repeatedly querying idiosyncratic items. ActiveUltraFeedback cuts preference data by up to six times using epistemic-network uncertainty, but it does not separate annotator noise from model uncertainty. The strongest constraint is Random Is Hard to Beat. It finds that uncertainty-based active preference selection gives negligible gains over random selection with modern pretrained models and costs about 20.2 times more compute. Any proposal here must beat random selection cheaply, or it has no reason to exist.

The closest bandit work, LLM-as-a-Judge on a Budget (ROBIN-HOOD) and Instance-Optimal Estimation with Multiple LLM Judges (Est-IVWE), allocates repeated judge queries toward high-variance items in order to estimate mean scores. That is the right objective for estimation, but it is the wrong objective for deciding which human labels will improve a judge. Under variance-seeking allocation, the most ambiguous items attract the most budget, which is exactly the behavior we want to avoid when buying training labels.

Research Program

The judge is an open-weight model fine-tuned with the distributional loss of Chen et al. On criteria where full fine-tuning is too costly, it is a frozen model with a small calibration head over its rating-token probabilities. We partition the pool into $K$ slices using metadata that is already available, namely criterion, source system or prompt category, and length bucket, plus embedding clusters. Annotation proceeds in rounds of $B$ labels. When a label for an item in slice $k$ arrives, we first record the current judge's cross-entropy on it, $\ell$, before training on it. This "predict-then-label" loss is an unbiased sample of slice-$k$ held-out loss and costs no extra annotation. The learning progress of slice $k$ at round $t$ is

$$\mathrm{LP}_k(t) = \bar{\ell}_k(t-w) - \bar{\ell}_k(t),$$

where $\bar{\ell}_k(t)$ is the average predict-then-label loss on slice $k$ over a sliding window ending at round $t$, and $w$ is the window lag. Cross-entropy against individual human ratings is bounded below by the entropy of the human rating distribution. On an irreducibly ambiguous slice, $\bar{\ell}_k$ therefore plateaus at that entropy and $\mathrm{LP}_k$ goes to zero, which is the annotation analogue of LPM's zero-equivariance. Slices are chosen with an upper-confidence rule on $\mathrm{LP}_k$, using a count-based bonus. Within a slice, items are drawn uniformly at random, so that we do not reintroduce item-level uncertainty sampling.

We will also test one extension. Each round may "deepen" an existing item by adding another annotator, or "widen" by labeling a new item. Deepening is treated as its own arm per slice, so the rule can discover where extra annotators sharpen the target distribution.

The known risk is LPM's "boredom trap." A slice the judge is too weak to learn quickly also shows near-zero progress and will be abandoned even though it is learnable. We address this with the optimism bonus and a minimum revisit rate, and we measure the risk directly rather than assuming it away.

Baselines

The naive reference is uniform random selection of items under the same budget, fine-tuning recipe, and seeds. Given Random Is Hard to Beat, this is a genuinely strong baseline, not a straw man.

The strongest existing selection method for judge calibration is the uncertainty-plus-diversity sampler of Active-Sampling-based Prompt Optimization. It reports reaching its SummEval MSE target with about 16% of training labels, against 32–50% for alternatives. We reimplement its convex selection objective on the same judge and slices and give it the same number of rounds and the same hyperparameter search budget, about twenty configurations per method chosen on a separate development split. Beating it means reaching its final held-out KL with at most 70% of its labels, or obtaining lower KL at equal budget with non-overlapping seed intervals, on at least two of the three datasets.

The second baseline adapts the variance-seeking allocation of ROBIN-HOOD to human labels. It allocates annotations to slices in proportion to an upper bound on human rating variance. For judge-query allocation, ROBIN-HOOD's own reported result is a worst-case correctness error of 0.300 at 50k queries, against 0.451 for uniform allocation, with GPT-4.1 nano on HelpSteer2. This number concerns estimation, not training, and we cite it only to show that the allocator is strong at its own task.

The third baseline is an epistemic-uncertainty sampler in the style of mutual information between predictions and model parameters. It uses an ensemble of five judge heads and subtracts expected entropy from total entropy. This is the most principled competitor and is discussed below.

Two ablations isolate the key ingredient. The "surprise" ablation selects slices by current loss $\bar{\ell}_k(t)$ rather than its decrease; this is the curiosity-by-prediction-error design that falls for the noisy TV. The "scalar target" ablation keeps progress-based selection but trains on the majority label, which removes the entropy floor. An oracle that uses true held-out progress computed from labels it does not pay for gives an upper bound.

Why not just…?

Why not just use random sampling, since it is known to be hard to beat? It may win, and that is the first thing the experiments must settle. The finding in Random Is Hard to Beat concerns item-level entropy selection for policy training. Our method differs in two ways that address its failure: selection happens at slice level, and the score costs one forward pass per labeled item, not 20 times the compute. If random sampling matches us on clean pools and loses only when the share of ambiguous items is high, that is the predicted pattern. If it matches us everywhere, the proposal fails.

Why not just filter out high-disagreement items and then run uncertainty sampling? A filter needs a disagreement estimate before labels exist, and the estimate is unreliable exactly where it matters. Conformal set width correlates with human-LLM error at only $r_s = +0.576$ (Diagnosing LLM Judge Reliability). A fixed threshold also cannot tell "ambiguous but partly learnable" from "pure noise." We include filter-then-uncertainty as an extra baseline with a tuned threshold. If it ties our method, the simpler recipe should be recommended instead.

Why not just use an epistemic mutual-information estimate, which already separates disagreement from model uncertainty? In principle it does, and this is the real threat. In practice it depends on the ensemble reflecting the posterior, and the evidence in Beyond Markovian that lightweight LoRA ensembles fail to do so suggests it may not. Measured progress needs no calibrated posterior. The head-to-head on the noise-injection test below decides this. If ensemble mutual information spends as little budget on noise slices as our method does and reaches equal KL, our contribution reduces to a cheaper approximation, and we will report it that way.

Proposed Experiments

We run pool-based simulations in which "annotation" reveals held-out human ratings. The datasets are SummEval (four criteria, using its multiple ratings per summary), SNLI and MNLI (multiple annotations per item), and HelpSteer2 (four attributes). The judges are Qwen-2.5-7B and Llama-3.1-8B. Budgets are 5%, 10%, 20%, and 40% of the pool, with five seeds each. The metrics are held-out KL divergence to human distributions, majority-label accuracy, conformal set width at 90% coverage per criterion, and MSE on the Likert mean for comparability with the prompt-optimization baseline. No reported result was found on Lacuna for active selection measured by distributional KL on SummEval, so the comparison there is run fresh.

The decisive check is a noise-TV test. Into each pool we inject synthetic slices whose human ratings are resampled uniformly at random, making up 10%, 30%, and 50% of items. We measure the share of the budget each method spends on these slices and the resulting held-out KL on real slices. Learning-progress selection should keep spending on noise near its exploration floor as the noise share rises. Uncertainty and variance-seeking methods should spend roughly in proportion to it.

A second check targets the boredom trap. We construct "slow but learnable" slices by holding out a rare topic and verify that progress-based selection returns to them once early progress appears. We report how often it abandons them for good.

The third check asks whether the gains transfer. We compare RewardBench accuracy of judges trained on HelpSteer2 under each selection rule, as a sanity check against overfitting to the selected slices. ArmoRM-Llama3-8B's reported 89.0 gives context for the scale of these numbers, not a target.

The proposal succeeds if it beats random and the uncertainty-plus-diversity sampler at matched budgets once ambiguous items exceed a modest share of the pool, and spends clearly less on injected noise than ensemble mutual information at matched final KL. It fails if random sampling ties it across the board.