Spending Judge Calls Where Candidate Models Disagree: Label-Efficient Model Selection Under Domain Shift

When a model trained on one data distribution (the source) is adapted to a new, unlabeled one (the target), the result usually falls well short of an "oracle" model trained on real target labels. Part of that shortfall comes from the adaptation methods themselves. Part comes from something more mundane. With no target labels, practitioners cannot tell which checkpoint, learning rate, or adaptation method actually works on the target, so they pick using source validation and often pick badly. On Camelyon17, source validation selects an overfit checkpoint at 53% target accuracy, while the best available checkpoint reaches 73% (Pseudo-Label Validation for Unsupervised Domain Adaptation). This proposal targets that selection component of the gap. It uses an LLM-as-a-judge, meaning a large language model prompted to grade whether a prediction is correct, as a cheap, biased label source for the target. A few dozen human labels are used to correct the judge's bias.

The central observation is that model selection does not need the judge to be accurate everywhere. To decide whether checkpoint A beats checkpoint B, only the target inputs where A and B predict differently carry any information. The judge's errors elsewhere cancel exactly. So both judge queries and human labels should go to these disagreement points, and the judge's bias should be measured on those points specifically. The output is an estimator of risk differences between candidates. It is unbiased with respect to the human labels, and it should need several times fewer human labels than a human-only validation set to reach the same selection quality. The goal is to state and test exactly when that holds.

Core Claim

For a pool of candidate target models and an unlabeled target set, we estimate each pairwise risk difference with a control-variate correction. A judge-imputed estimate is computed over many unlabeled points. A correction term is then added, computed on a small human-labeled sample. It measures how far the judge-imputed difference deviates from the true difference on those points.

Write $f_j, f_k$ for two candidates, $q(x)$ for the judge's soft label on input $x$ (averaged over repeated judge queries), $\ell$ for the loss, $U$ for the $N$ unlabeled target points and $L$ for the $n$ human-labeled ones. Define $d_q(x) = \ell(f_j(x), q(x)) - \ell(f_k(x), q(x))$, and let $d_y(x,y)$ be the same quantity computed with the true label $y$. The estimator is

$$\hat\Delta_{jk} = \frac{1}{N}\sum_{x \in U} d_q(x) + \frac{1}{n}\sum_{(x,y)\in L} \big(d_y(x,y) - d_q(x)\big).$$

The construction is a standard control variate, and we do not claim it as new. Two properties make it useful here. First, under 0-1 loss both $d_q$ and $d_y$ are zero wherever the two candidates agree, so all variance lives on the disagreement set. Second, the estimator beats the human-only estimate exactly when the judge-imputed difference correlates strongly enough with the true difference on that set. That condition can be checked from the calibration sample itself.

What is new is the combination of three things. We restrict judge queries, repeated judge sampling, and human labeling to disagreement regions between candidates. We give a test, estimable from the calibration sample, for when the judge helps rather than hurts. And we show empirically, on text shifts where unsupervised adaptation has failed, that this closes most of the selection component of the oracle gap.

Relation to Prior Work

Two papers motivate the proposal by showing the selection problem is real and partly solvable.

Pseudo-Label Validation for Unsupervised Domain Adaptation replaces target labels with soft labels from a source-trained "imputer." It proves the imputer should be deliberately undersmoothed, because the imputer's bias accumulates across points while its variance averages out. That argument is exactly why we add a human correction. Bias is the irreducible part, and a source-trained imputer becomes systematically biased under severe shift. We differ in three ways: our imputer is an external judge, we remove its bias with human labels rather than tolerate it, and we target the disagreement set rather than the whole target.

Extending the WILDS Benchmark for Unsupervised Adaptation shows that adaptation methods rarely beat plain empirical risk minimization (ERM, ordinary training on source labels) on text tasks such as CivilComments and Amazon. On text, then, choosing well among imperfect candidates may matter more than designing yet another adaptation method.

Three papers supply the method.

Noisy but Valid estimates an LLM judge's true- and false-positive rates on a human calibration set. It then derives a "green region" where the judge adds statistical power over the humans alone. Our usefulness condition is the analogue for comparing two models instead of certifying one. That paper only tests whether a single model's binary safety failure rate is below a threshold, and never feeds the judge into model selection.

LLM-as-a-Judge on a Budget shows that judge-score variance differs widely across inputs. Its bandit allocator, ROBIN-HOOD, halves the cost of estimating judge means. We reuse its variance-aware allocation for repeated judge queries, but only on the disagreement set. That paper also assumes the judge's mean is the quantity of interest. We explicitly do not, since the judge's bias relative to humans is the thing we correct.

LabOR and Discrepancy-Based Active Learning for Domain Adaptation both send scarce labels to uncertain or uncovered target regions, LabOR by where two classifiers disagree. We borrow the disagreement heuristic, but use the labels to select among models rather than train them, and we assume a noisy rather than perfect oracle.

One paper constrains the design. Limitations of refinement methods for weak to strong generalization proves that training on refined weak labels leaves irreducible error. This is why we do not claim that judge labels used for training can reach the oracle. We use the judge only inside an estimator whose bias is corrected by humans, so its residual error turns into variance rather than bias.

Research Program

We build the candidate pools on the WILDS text datasets CivilComments and Amazon, which come with curated unlabeled target data. Each pool contains roughly 50 to 100 models. They come from ERM, adversarial feature alignment, correlation alignment, pseudo-labeling, and Noisy Student, each swept over checkpoints and learning rates. The pool also includes one student trained directly on judge labels, so the procedure can decide whether distilling the judge was worth it.

The judges are the open models used in Noisy but Valid (Qwen2.5, Mistral-7B, LLaMA-3) and the two used in the Budget paper (LLaMA-3.1-8B-Instruct, GPT-4.1-nano). This range spans weak to fairly strong judges. Each judge receives the input and the candidate label and returns a chain-of-thought justification followed by a correctness probability. Human labels are simulated by revealing held-out target labels, at budgets $n \in \{0, 25, 50, 100, 200, 500\}$.

The procedure runs in four steps. It first identifies the disagreement points across the leading candidates. It then allocates repeated judge queries over them using ROBIN-HOOD-style variance bounds. Next, it draws the human sample from the same disagreement region, weighted by how much that region contributes to the uncertainty in the remaining comparisons. Finally, it compares candidates with $\hat\Delta$, eliminating weaker ones over successive rounds.

The analysis has three parts. We derive the variance of $\hat\Delta$ against the human-only estimator, which yields the break-even judge–truth correlation on the disagreement set. We check whether the plug-in estimate of that correlation predicts, before any selection happens, whether the judge will help. And on Camelyon17 we substitute the source-trained imputer from Weill and Wang for the LLM judge. This tests whether the human correction and disagreement targeting add value even when the label source is not an LLM.

Baselines

The naive reference is source-validation selection: pick the candidate with the lowest loss on labeled source data. This is the default practitioners fall back on, and it fails on Camelyon17 (53% against a 73% oracle, per Weill and Wang). It uses the same pool and zero human labels.

The strongest competitor on the question we actually ask is human-only target validation: the same $n$ human labels, drawn uniformly from the target, used to estimate each candidate's risk directly. It is unbiased, needs no judge, and is what a careful practitioner would do. We run it with the identical pool and budget. Beating it means lower selection regret (the oracle-best candidate's target score minus the selected candidate's) at equal $n$, or equal regret with at most half the labels. We also run a stronger version that draws its $n$ labels from the disagreement set. This separates the value of targeting from the value of the judge.

The strongest existing label-free method is pseudo-label validation with a source-trained, undersmoothed imputer (Pseudo-Label Validation for Unsupervised Domain Adaptation). On Camelyon17 it reports 73% selected accuracy, matching the oracle checkpoint. No reported result found on Lacuna for this method on CivilComments or Amazon, so we run it ourselves with its recommended undersmoothing.

Two ablations isolate the key ingredients.

Judge-only selection applies the uncorrected first term of $\hat\Delta$, with $n=0$. Its gap to the full method measures how much the judge's bias would have misled selection.

Uniform judge sampling gives a fixed number of judge queries to every point. Against it, we report the Budget paper's correctness numbers on HelpSteer2 (worst-case absolute error at 50k queries with GPT-4.1-nano: 0.451 uniform against 0.300 for ROBIN-HOOD) as the expected size of the allocation effect. We do not use these as a target, since that is a different task.

Every method receives the same pool, the same judge-query budget where judges are used, and the same human labels where humans are used.

Why not just…?

Why not just label a few hundred target examples and validate on them? This is the real threat, and the experiment is built around it. With 500 uniform labels, human-only validation may already select near-oracle models, leaving the judge nothing to add. Our claim is restricted to the low-budget regime and to fine-grained comparisons among close candidates, where differences hinge on a small disagreement set that uniform sampling rarely hits. The disagreement-targeted human-only baseline tests whether the judge contributes anything beyond smarter sampling. If that baseline matches our method, the honest conclusion is that targeting, not judging, does the work, and we will report it that way.

Why not just deploy the LLM zero-shot, or distill it into the target model? If the judge classifies the target better than every adapted model, selection is moot. We report each judge's zero-shot target accuracy as a reference line. The distilled student sits inside the pool, so the procedure itself tells us when distillation wins. The weak-to-strong result (Limitations of refinement methods) suggests the student will inherit the judge's systematic errors. Selection is still needed whenever deployment rules out calling a large model on every input, or whenever the judge is only moderately accurate, which is the common case for weaker judges.

Why not just use the judge's labels uncorrected, like a better pseudo-label imputer? Weill and Wang's own analysis says imputer bias accumulates rather than averaging away. An LLM judge's bias on CivilComments, where its notion of toxicity may differ from the annotators', is exactly this kind of error. The judge-only ablation measures how often this bias flips a ranking. If it never does, the human correction is unnecessary and the method reduces to a cheaper pseudo-label validator. That would also be a useful finding.

Decisive Experiments

The primary check plots selection regret against human budget on CivilComments and Amazon. Each WILDS dataset is scored with its official target metric and with average accuracy. Curves cover all five judges and all baselines, averaged over 20 resamplings of the human set. The method succeeds if, for judges inside the predicted usefulness region, it reaches the regret of 200-label human-only validation using 50 or fewer labels. It must also never do worse than human-only validation by more than the estimator's stated confidence width.

The second check tests whether the usefulness condition is predictive. On each dataset–judge pair, the correlation estimated from the first 25 labels should sort the cases where the judge helped from those where it hurt. Agreement on fewer than three quarters of pairs would falsify the claim.

The third check is the Camelyon17 imputer substitution. Adding 25 to 50 corrected labels to pseudo-label validation should not harm its reported 73% selection, and should rescue it on harder shifts.

Throughout, we report how far selection alone moves the chosen model toward the oracle, separately from any adaptation gain. This shows how much of the "oracle gap" on text was actually a validation gap.