Many adaptive data-collection systems work the same way. They pick the next experiment, comparison, or label to maximize the determinant of the Fisher information matrix, a summary of how much the data pins down the model's parameters. This is D-optimal design. Ranking LLMs in an arena, choosing which subsets of responses a human should rank, and choosing which patients or compounds to measure all use it. In a logistic or Bradley-Terry model, the information a candidate point contributes depends on the model's current guess of the parameters. That dependence is the familiar "catch-22" of nonlinear design. This note argues that the catch-22 has a specific, quantifiable form when the number of parameters is a meaningful fraction of the number of labels, which is exactly the early phase of any active loop. In that regime the fitted parameters are systematically too large in magnitude, so the model overstates its confidence on new points. A plug-in D-optimal rule therefore misjudges which candidates are informative.
The proposal is to correct this with approximate leave-one-out estimation, which predicts from a single model fit what the model would have output on each labeled point had that point been held out. Those held-out predictions show, from data already collected, how the model's scores on unseen points relate to actual outcomes. Since every candidate in the pool is by definition unseen, the same relation can recalibrate the per-candidate information weights before the determinant is maximized. In two sentences: leave-one-out scores are honest samples of how the model behaves on new points, so they can be used to correct the information each new point is predicted to provide. Doing so should make adaptive D-optimal design spend its early budget on genuinely informative queries rather than on the boundary of an overconfident model.
Core Claim
Take a logistic model with labeled data $(x_i, y_i)$, $i = 1, \dots, n$, parameter dimension $p$, and fitted parameter $\hat\theta$. Standard adaptive D-optimal design scores an unlabeled candidate $x$ by how much it would increase $\log\det I$. Here $I = \sum_i w_i x_i x_i^\top$ is the current Fisher information, and the plug-in weight is $w(x) = \sigma'(x^\top\hat\theta)$, with $\sigma$ the logistic function. When $\kappa = p/n$ exceeds roughly 0.1, SLOE documents two effects. The maximum-likelihood estimate behaves like an inflated, noisy copy of the truth, roughly $\alpha\theta^*$ plus Gaussian noise with $\alpha > 1$. Standard covariance estimates also understate uncertainty, so nominal 90% intervals cover only 60–70% of the time. Plug-in weights inherit both errors. Inflated margins make $\sigma'$ too peaked, concentrating predicted information on points near the estimated decision boundary. Ignoring the noise also overstates how informative those points really are.
We replace the plug-in weight with a calibrated one,
$$ w_{\text{cal}}(x) = \mathbb{E}_{u \sim \mathcal{N}(x^\top\hat\theta/\hat\alpha,\ \hat\tau^2)}\big[\sigma'(u)\big], $$
where $\hat\alpha$ estimates the inflation factor and $\hat\tau^2$ the variance of the true margin given the estimated one. Both are fitted on the approximate leave-one-out margins $x_i^\top\hat\theta_{(-i)}$ of the already-labeled points, not on their in-sample margins. The greedy acquisition rule then becomes
$$ x_{\text{next}} = \arg\max_{x}\ \log\big(1 + w_{\text{cal}}(x)\, x^\top I_{\text{cal}}^{-1} x\big), $$
where $I_{\text{cal}}$ is built from the labeled points using weights computed from their leave-one-out margins. This is the standard rank-one determinant update, the same Woodbury-style bookkeeping that DopeWolfe and the arena work use, so the method adds almost no cost.
The claim that would be new is the following. In the moderately high-dimensional regime, D-optimal designs computed with leave-one-out-calibrated weights reach a target parameter error or ranking accuracy with measurably fewer labels than plug-in D-optimal designs. They also beat designs that average plug-in weights over a Laplace posterior. The advantage should shrink to zero as $\kappa \to 0$. That last prediction is what makes the claim falsifiable rather than a generic "better uncertainty helps."
Relation to Prior Work
The proposal reuses the mechanism of the approximate leave-one-out literature. Rad and Maleki's error bounds show that a single Newton step from the full fit, $\hat\theta_{(-i)} \approx \hat\theta + H^{-1}\nabla\ell_i(\hat\theta)$, tracks exact leave-one-out with error vanishing even when $p$ is proportional to $n$. That result justifies computing all $n$ held-out margins in one pass. SLOE goes further for unregularized logistic regression: it uses Sherman-Morrison updates to turn those held-out margins into estimates of the inflation and noise parameters that drive high-dimensional bias. Our estimators for $\hat\alpha$ and $\hat\tau$ come directly from that construction.
Both papers use the corrections only for inference and risk estimation on a fixed dataset. Neither feeds them back into which data to collect. Iterative Approximate Cross-Validation extends leave-one-out tracking to non-converged gradient-descent iterates. We use it as an engineering option for refits that are not run to convergence between acquisitions, but it is likewise purely passive.
The design side supplies the application regime and the main competitors. Fewer Battles, More Gain applies D-optimality to Bradley-Terry arenas. It shows that D-optimality avoids A-optimality's tendency to over-compare against one anchor model by favoring roughly even matchups. DopeWolfe scales D-optimal subset selection under Plackett-Luce to trillions of candidate subsets. Both plug the current estimate directly into the Fisher matrix. Both also start from a random warm-up because the information matrix is singular at first. The early stage is precisely where $p/n$ is large and plug-in weights are least trustworthy, so our correction targets the stage these systems handle with an ad hoc fix.
IBOSS frames subdata selection as D-optimal design, but in a linear model where information does not depend on parameters. Our problem therefore does not arise there, and IBOSS is not a competitor in the nonlinear case.
The closest connecting work is Optimal Subsampling with Influence Functions. It samples points in proportion to the norm of $H^{-1}\nabla\ell_i$ computed at a pilot estimate. It shares the leave-one-out quantity, but it uses it as a one-shot importance weight under a trace (A-type) criterion. The approach is open loop, and it requires labels for every candidate to compute gradients. Our setting differs on each point: candidates are unlabeled, acquisition is sequential, the criterion is the determinant, and leave-one-out quantities are computed only on labeled points and used to calibrate predictions for unlabeled ones.
Two papers constrain the design. SLOE's estimators require the maximum-likelihood estimate to exist, and early active data is often linearly separable. We therefore work with a lightly ridge-regularized fit and use the regularized leave-one-out formulas, which the Rad-Maleki analysis covers. ProxACV warns that Newton-step approximations degrade when curvature is low. We check this directly, because the very early rounds of an active loop are low-curvature.
Research Program
The first component is the estimator. For logistic and Bradley-Terry models with ridge penalty $\lambda$, we compute held-out margins for all labeled points in one pass using the Newton-step formula, which needs only the diagonal of a hat-type matrix. We then estimate $\hat\alpha$ and $\hat\tau$ in two ways.
The first way follows SLOE's moment construction, which is principled under roughly Gaussian features. The second fits $\hat\alpha$ and $\hat\tau$ by maximizing the likelihood of the observed labels given the held-out margins under the Gaussian-convolved logistic model above. From binary labels alone, this second fit identifies only a combination of $\hat\alpha$ and $\hat\tau$. We break the tie with a variance estimate taken from the spread between full-fit and held-out margins, which scales with estimator noise.
We treat this identification issue as a real risk. If the two routes disagree badly on non-Gaussian data, that is itself a finding about when the correction can be trusted.
The second component is the acquisition loop. We plug $w_{\text{cal}}$ into greedy D-optimal selection for single points and arena pairs, and into DopeWolfe's randomized Frank-Wolfe for Plackett-Luce subsets. Calibration is recomputed after each batch at cost linear in $n$ beyond the existing Hessian factorization.
The third component is analysis. In the proportional asymptotic model used by SLOE, the inflation and noise parameters are deterministic functions of $\kappa$ and signal strength. That makes it possible to compute, as a function of $\kappa$, the gap between the plug-in criterion and the oracle criterion that uses $\theta^*$. It also makes it possible to compute how much of that gap the calibrated criterion closes. We aim for a characterization showing the plug-in design's regret over the oracle growing with $\kappa$ and the calibrated design's regret staying bounded. That result would explain why the effect disappears in classical $n \gg p$ settings.
Baselines
The naive reference is uniform random acquisition: draw candidates, pairs, or subsets uniformly, with the same model, ridge penalty, and refitting schedule. Every adaptive method must beat it. The literature says this margin is large. DopeWolfe reports an order-of-magnitude reduction in the samples needed to reach a target ranking accuracy on Nectar relative to random sampling. The arena paper reports that D-optimal selection beats random and the Ada-Pair heuristic, but the file gives no numeric table for it (no reported numeric result found on Lacuna for Chatbot Arena ranking accuracy).
The strongest existing competitor is plug-in D-optimal design as practiced by the arena paper and by DopeWolfe. That is the identical greedy or Frank-Wolfe determinant maximization with weights $\sigma'(x^\top\hat\theta)$ at the current ridge estimate. It is the strongest because it is the deployed method, and our proposal changes exactly one thing about it. We run it with the same warm-up size, candidate pool, ridge penalty, and refit schedule. We give it the same budget for tuning $\lambda$ that our method gets, selected on a separate pilot problem.
Beating it means a statistically significant reduction, across at least 20 random seeds, in labels-to-target on parameter error or Kendall tau at $\kappa \geq 0.2$, with no loss at small $\kappa$. As a benchmark for scale, DopeWolfe's order-of-magnitude gain over random on Nectar is the published reference point. We expect our gains over plug-in D-optimal to be much smaller and concentrated in early rounds.
The key ablation is Laplace-averaged D-optimal design. It uses the same Gaussian-convolved weight, but centers the convolution at the uncorrected margin $x^\top\hat\theta$ and takes its variance from the plug-in inverse Fisher matrix rather than from leave-one-out quantities. This isolates the claimed ingredient: if Laplace averaging matches our method, the gain comes from averaging over uncertainty, not from leave-one-out calibration.
We also run a deflation-only variant, which divides by $\hat\alpha$ but sets $\hat\tau = 0$, and a variance-only variant, which keeps $\hat\alpha = 1$. These separate the two corrections.
Finally, we report two reference designs. One is a sequential version of influence-function sampling (Ting and Brochu), which uses expected gradient norms under the current model, since candidate labels are unknown. The other is an oracle D-optimal design that uses the true $\theta^*$ on synthetic data, which sets the ceiling.
Why not just…?
Why not just use Bayesian D-optimal design with a Laplace posterior? Averaging the Fisher weight over a posterior is the textbook answer to the catch-22, and it is the most serious threat to this proposal. The problem is that in the proportional regime the Laplace posterior is centered at an inflated mean. Its covariance is the inverse Fisher matrix, which SLOE shows understates the true variability, so averaging blurs a curvature profile that is shifted in the wrong direction. The Laplace-averaged ablation above is designed to settle this. If it matches the calibrated method at $\kappa \geq 0.2$, we will report that leave-one-out calibration adds nothing beyond posterior averaging, and the proposal fails.
Why not just tune the ridge penalty more aggressively so the estimate is not inflated? Shrinkage does pull margins toward zero. But it shrinks every coordinate by a factor set to minimize prediction risk, which is not the factor that makes curvature weights correct. Choosing that penalty well in high dimensions itself requires cross-validation, which is again a leave-one-out computation. We test this by giving the plug-in baseline a ridge penalty chosen by approximate leave-one-out risk at every round. If that tuned plug-in baseline matches our method, the simpler fix wins, and we will say so.
Why not just collect a larger random warm-up until $n \gg p$, then switch to D-optimal design? This removes the bias by brute force, but it spends the budget where adaptive design matters most. On arenas with hundreds of models and on Plackett-Luce problems with large $N$, the warm-up needed to make $\kappa$ small is a substantial fraction of the total budget. We sweep warm-up size for the plug-in baseline and report the best setting found in hindsight. Our method must match or beat even that best-in-hindsight setting without the extra tuning.
Proposed Experiments
The first testbed is synthetic logistic regression following SLOE's setup, with $\kappa$ swept from 0.02 to 0.5 at the point where acquisition begins and with both Gaussian and heavy-tailed features. Ground truth is known there, so we can measure the mechanism directly. Before running any active loop, we compare plug-in and calibrated weights against the true expected Fisher weight $\sigma'(x^\top\theta^*)$ on held-out candidates. The idea requires that calibrated weights be measurably closer at large $\kappa$. If they are not, the design experiments are moot.
We then run full active loops and report parameter error, held-out log-loss, and labels-to-target, plotted against $\kappa$. The decisive prediction is a gap over plug-in D-optimal design that grows with $\kappa$ and vanishes as $\kappa \to 0$.
The second testbed is an arena simulation built from Chatbot Arena outcomes, following the arena paper. Bradley-Terry strengths are fit on the full data and treated as ground truth. Pair acquisition is then replayed by sampling outcomes from those strengths, which puts the early regime at large effective $\kappa$. Metrics are Kendall tau to the ground-truth ranking and variance of strength estimates against the number of battles.
The third testbed is Plackett-Luce subset acquisition on the Nectar and Anthropic preference data and on BEIR-COVID and TREC-DL, inserting calibrated weights into DopeWolfe. The metric is the Kendall tau ranking loss used in that paper, and we report wall-clock time to confirm the overhead is negligible.
As a real-data check with non-Gaussian, discrete features, we run pool-based active labeling on UCI Heart Disease, which SLOE also uses, at reduced training sizes chosen to place $\kappa$ above 0.1.
Two further checks guard against over-claiming. First, at each round on synthetic data, we compare approximate leave-one-out margins with exact refits. This lets us report where Newton-step error is large, which ProxACV predicts will happen in low-curvature early rounds, and whether those rounds are where the method fails. Second, we report results for both estimators of $\hat\alpha$ and $\hat\tau$ on non-Gaussian data, so that any dependence on SLOE's Gaussian-feature assumption is visible.
The proposal succeeds if calibrated weights are closer to the truth and the active-loop gain over plug-in D-optimal design grows with $\kappa$ while surviving the Laplace-averaged and tuned-ridge ablations. It fails, and is reported as failing, if either ablation closes the gap.
References
- SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression
- Comparing Few to Rank Many: Active Human Preference Learning Using Randomized Frank-Wolfe Method
- Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions
- Iterative Approximate Cross-Validation
- Fewer Battles, More Gain: An Information-Efficient Framework for Arena-based LLM Evaluation
- Information-Based Optimal Subdata Selection for Big Data Linear Regression
- Optimal Subsampling with Influence Functions
- Approximate Cross-validation: Guarantees for Model Assessment and Selection