Fewer, Better Comparisons: Noise-Calibrated Adaptive Pairwise Refinement

Some prediction problems are easier to judge by comparison than by absolute value. A chemist, a human rater, or a large language model may be unable to say "this molecule's solubility is −3.2" but can often say "this molecule is more soluble than that one." Post Hoc Regression Refinement via Pairwise Rankings (RankRefine) turns that observation into a method. It takes a regressor's first guess, asks a ranker to compare the query against 20 labeled reference items, and fuses the two estimates. It reports gains even when the ranker is right only about 55% of the time. The cost is that every refinement makes 20 sequential calls to a slow, expensive, noisy judge. This is the same latency problem that draft-critique-revise loops have: each extra pass improves quality, but the passes add up.

We propose to spend comparisons adaptively. The method first measures how noisy the ranker is by showing it pairs of reference items whose true order is already known. It then uses that noise measurement to pick, one batch at a time, the reference that is most informative for the current query, and it stops once another comparison is no longer worth its cost. The central claim is that the noise model is what makes adaptivity safe. With a noisy ranker, naive adaptive rules chase the closest, hardest-to-judge pairs and stop too early on overconfident estimates. Measuring the ranker's error rate as a function of how far apart two items are prevents both failures.

Core Claim

RankRefine treats the ranker's outputs as draws from a Bradley–Terry model, which is a logistic model where the probability that item A beats item B grows with the gap in their true values. It sets the precision of its rank estimate from the curvature of that likelihood. For a noisy ranker this assumption fails in a specific way. Its errors look less like a softened logistic curve and more like random flips. The flips are also probably most frequent on pairs whose true values are close. Boosting the Area under the ROC Curve shows that pairwise signals survive noise that flips labels independently at a rate below one half. Its guarantee assumes that the flip rate does not depend on the instance, and we expect that assumption to fail most badly on close pairs. Close pairs are exactly the ones an information-seeking selector wants to query. This produces a trade-off that fixed-budget RankRefine never encounters, and that an off-the-shelf adaptive design gets wrong.

We model the ranker's answer to "is the query larger than reference $j$?" as

$$P(\text{yes}\mid y) = \eta(\Delta) + \big(1-2\eta(\Delta)\big)\,\sigma\!\left(\frac{y-y_j}{s}\right),$$

where $y$ is the query's unknown target, $y_j$ is the reference's known label, $\sigma$ is the logistic function, $s$ is a scale, and $\eta(\Delta)$ is a lapse rate. The lapse rate is the probability that the ranker's answer is simply flipped, and it depends on the gap $\Delta=|y-y_j|$. Both $s$ and $\eta(\cdot)$ can be fit without touching any query, because comparisons between two references have known answers. Under this model, the information a single comparison carries about $y$ peaks at a moderate gap rather than at zero gap. Very close pairs are ambiguous by nature and are also the most often flipped, while very distant pairs are predictable and teach little. The selector therefore queries the reference that maximizes expected information under the current posterior over $y$. It stops when the expected reduction in the fused estimate's error from one more batch of $b$ comparisons falls below a per-call cost $c$. Fused intervals are computed from this lapse-aware likelihood instead of from the raw Bradley–Terry curvature.

Research Program

The work has three parts. The first is noise calibration. For each task and ranker, we collect a small offline set of reference–reference comparisons, stratified by label gap. We fit $s$ and a monotone $\eta(\Delta)$ to these comparisons and test whether the lapse rate really rises for close pairs. If it does, the independent-noise assumption fails in the direction we predict. If it does not, the design reduces to a constant-lapse model, which is simpler but is still a finding.

The second part is batched adaptive selection with a stopping rule. Wall-clock latency depends on the number of rounds, and monetary cost depends on the number of calls. Each round therefore issues $b$ comparisons in parallel, chosen jointly to cover different gaps, and we report both quantities. The stopping rule is posed as a cost-sensitive decision and not as a fixed variance threshold. We also check that the resulting intervals keep their stated coverage even though the number of comparisons is chosen after looking at the data. Choosing when to stop based on the data seen so far is known to bias naive intervals.

The third part is transfer. We ask whether a noise curve calibrated on one assay transfers to a related assay, and whether one fitted for a language-model ranker behaves like one fitted for human raters. This decides whether the calibration step must be repeated for every deployment.

The latency literature motivates this framing. DEER stops chain-of-thought reasoning once it becomes confident and reports 19.1% to 80.1% fewer generated tokens. Repoformer skips retrieval when retrieval would not help and reports up to 70% speedup. Both depend on scalar confidence thresholds that the research file flags as prone to miscalibration. POPE documents "yes" rates near 99% in some vision-language settings, which is a warning that a judge's own confidence cannot serve as its noise estimate. We take the adaptive-computation idea from these papers, but we calibrate the stopping signal against pairs whose answers are known instead of against the model's self-reported confidence.

Baselines

The do-nothing reference is the base regressor alone, with no comparisons. Every refinement method must beat it, and its error sets the scale for every relative gain we report.

The strongest existing method is RankRefine exactly as published: $k=20$ references, a Bradley–Terry fit, and inverse-variance fusion. The paper reports up to a 10% relative reduction in mean absolute error (MAE) on TDC ADME, gains over the base regressor at ranker accuracy around 0.55, and a 5% error reduction in its UTKFace human study (RankRefine). We will rerun it with the same base regressors, reference pools, rankers, and prompts. We will also run it at every fixed budget from 1 to 20, which traces its MAE-versus-calls curve.

Beating RankRefine has two parts. The adaptive method's MAE must be statistically indistinguishable from RankRefine at $k=20$ while using a median of at most 40% of the calls. And at every matched average budget, its MAE must be no worse than fixed-budget RankRefine. We set this pre-registered bar ourselves; it is not a reported figure.

Two ablations isolate the key ingredient. The first uses adaptive selection and stopping under the plain Bradley–Terry model, with no lapse term. It tests whether adaptivity alone is enough. The second uses the lapse-aware likelihood with randomly chosen references and a fixed budget. It tests whether better fusion alone is enough. All methods share the same total tuning budget, and the stopping cost $c$ is swept, not tuned per dataset.

Why not just…?

Why not just issue all 20 comparisons in parallel? This removes sequential latency entirely, and against the wall-clock metric it is a real competitor. It does nothing for cost, however, and cost is what binds when the ranker is a paid API or a human expert, as in RankRefine's UTKFace study. We will report calls and rounds separately. If the only goal is wall-clock time and cost is free, the parallel baseline wins, and we will say so.

Why not just use a smaller fixed $k$ with random references? This is the fixed-budget curve described in the Baselines section. If the MAE-versus-$k$ curve saturates early for every query, adaptivity has nothing to add. Our hypothesis is that the saturation point varies widely across queries. Queries whose first guess is already good, or whose neighbors are well separated, should need few comparisons, while ambiguous queries need many. A per-query analysis of how many calls were actually used will settle this directly.

Why not just stop once the rank variance drops below the regressor's variance? This is the obvious rule, and it is the one the research file suggests. Its variance estimate comes from a likelihood that ignores flips, so we expect it to be overconfident exactly when the ranker is noisy, and therefore to stop too early. The no-lapse ablation implements this rule, and interval coverage under synthetic flip noise will show whether its estimates are overconfident.

Proposed Experiments

We will use the nine TDC ADME regression datasets and UTKFace age estimation, following RankRefine's low-data setup. We will test three families of rankers. The first is a language-model ranker (RankRefine used GPT-4o). The second is synthetic rankers built from true labels, with a constant flip rate set to give accuracies from 0.55 to 0.9. The third is synthetic rankers whose flips concentrate on close pairs, which isolates the instance-dependent failure that the independent-noise theory does not cover. Metrics are MAE, the number of ranker calls, the number of rounds, and the empirical coverage of 90% intervals.

Three checks decide the outcome. First, under gap-dependent synthetic noise, the no-lapse adaptive ablation should lose coverage while the full method keeps it. If both keep coverage, the noise model is unnecessary. Second, on the language-model ranker, the fitted lapse curve should rise for close pairs. If it is flat, the independent-noise assumption holds in practice and the design simplifies. Third, the call-saving bar from the Baselines section must hold on at least most of the ADME datasets at matched MAE. We are not proposing this as a general theory of stopping for generative refinement loops; if the approach succeeds here, the natural next test is critique-revise loops, where comparing a revision against its previous draft is the pairwise judgment.