Rank-Informed Neyman Auditing of AI-Delegated Decisions

Many deployed systems now split work between a model and people. The model handles the cases it is confident about, and humans review the rest. The cases the model handled alone are rarely checked carefully, yet someone must vouch for their error rate: an exam board, a pathology lab, or a data-labeling vendor. The usual practice is to send a random slice of automated decisions to human auditors. This proposal treats that audit as a sampling-design problem. Each automated decision is sent to an auditor with a probability chosen to make the error-rate estimate as precise as possible for a fixed number of human checks. The target is Neyman allocation, the classical rule that spends samples in proportion to each unit's standard deviation, with errors as the outcome. Confidence scores are often miscalibrated, so the rule does not trust their numerical values. It trusts only their ordering: it assumes that less confident decisions are more likely wrong, and it learns how much more likely from the audits as they arrive.

In two sentences: allocate a fixed human audit budget across automated decisions in proportion to each decision's estimated error standard deviation, where that standard deviation comes from a monotone (isotonic) fit of audited errors against the model's confidence rank. Because the method relies only on ordering, it keeps its efficiency when the model is over- or under-confident. A running rank correlation between confidence and audited error tells it when to fall back to uniform sampling.

Core Claim

Consider an automated stream of $T$ items. Let $y_i \in \{0,1\}$ mark whether item $i$ is wrong, which is observed only if a human audits it. Let $\hat e_i$ be a current estimate of its error probability. The model-assisted (difference) estimator of the stream's error rate is

$$\hat\theta = \frac{1}{T}\sum_{i=1}^{T}\Big(\hat e_i + \frac{A_i}{\pi_i}(y_i - \hat e_i)\Big),$$

where $A_i$ indicates that item $i$ was audited and $\pi_i$ is its audit probability. For a fixed expected budget $B=\sum_i \pi_i$, the variance of this estimator is minimized, up to the error in $\hat e_i$, by

$$\pi_i \propto \sqrt{e_i(1-e_i)},$$

capped at 1. This is Neyman's rule applied at the level of single items. It follows from the same Lagrangian argument that gives $n_k\propto\sigma_k$, because the variance of $y_i$ is $e_i(1-e_i)$.

The difficulty is entirely in $e_i$. Raw confidence values are unreliable error probabilities. Two Sides of Miscalibration shows that Vision Transformers and BERT-style models are systematically under-confident, and that global temperature scaling can make this worse. Uncertainty in Language Models: Assessment through Rank-Calibration shows that the ordering of uncertainty is a more stable and comparable property than its scale; for example, NLL is well rank-calibrated on TriviaQA with Llama-2-chat. Our claim is that rank calibration is exactly the property Neyman auditing needs. If error probability is monotone in confidence, then the isotonic regression of audited errors on confidence rank pools information across all audited items. That pooled fit estimates $e_i$ for unaudited items far faster than per-stratum variance tracking, and it is immune to monotone distortions of the confidence scale. A running Kendall's $\tau$ (the fraction of concordant minus discordant pairs) between confidence and audited error is used as a safety switch. When $\tau$ is near zero, the ordering carries no information, and the design shrinks toward uniform auditing.

Relation to Prior Work

Three papers supply the allocation machinery. CLIP-OGD tracks the Neyman ratio online with clipped gradient steps and achieves $O(\sqrt{T})$ Neyman regret, meaning the excess variance over the oracle allocation. Logarithmic Neyman Regret introduces ClipSMT, which plugs empirical second moments into the Neyman ratio with clipping $c_t=\tfrac12 t^{-1/3}$. It reaches logarithmic regret in stationary settings. Asymptotically Optimal and Computationally Efficient ATE Estimation shows that Neyman allocation is the small-interval limit of optimal sequential sampling. All three allocate among a handful of treatment arms using only outcome moments. None of them uses a per-item covariate such as model confidence. The file also notes that contextual, AIPW-style Neyman allocation lacks finite-sample guarantees. We reuse ClipSMT's clipping and second-moment logic, and we replace per-arm moments with a shape-constrained function of confidence rank. On the Complexity of A/B Testing motivates the work by proving that uniform sampling is suboptimal whenever variances differ, and the automated stream is a strongly heteroscedastic population.

The delegation literature motivates the problem but never addresses audit precision. Preventing Critical Scoring Errors in Short Answer Scoring and Balancing Cost and Quality rank short answers by posterior probability or Trust Score and automate the top fraction. Deep learning based semi-automated model can predict lineage in patients with pituitary neuroendocrine tumors uses a fixed 0.8 threshold on histology. CrowdAgent routes tasks among LLMs, small models and humans using heuristic agents. All of them assess the automated stream on a held-out labeled set. None says how a live deployment should spend reviewer time to certify that stream. Balancing Cost and Quality also reports that thresholds tuned on small development sets become unstable, which suggests that certification has to continue during deployment. Is the Most Accurate AI the Best Teammate? optimizes the delegation decision itself for team utility. We hold the delegation policy fixed and optimize only the audit. The incompatibility result in CLIP-OGD constrains the design: auditing for precision is a different objective from routing for reward, and so we keep the two separate.

Rank-calibration analysis enters as a modeling assumption and as a diagnostic, not as an evaluation afterthought. Uncertainty in Language Models: Assessment through Rank-Calibration defines rank-calibration error (RCE) offline and notes that its finite-sample behavior is unproven. We use the online analogue, Kendall's $\tau$ on audited pairs, to control how far the design departs from uniform. The file names a known weakness of rank methods: they discard magnitude, but Neyman needs ratios of standard deviations. Isotonic regression addresses this directly. It takes the ordering from the confidence ranks and the magnitudes from audited outcomes.

Research Program

The method, called RINA (Rank-Informed Neyman Auditing), runs as a stream. Each arriving automated item receives a confidence score. The current isotonic fit maps the score's rank to $\hat e_i$. The audit probability is set to $\pi_i \propto \sqrt{\hat e_i(1-\hat e_i)}$ and normalized to the remaining budget. It is then mixed with uniform probability using weight $w_t = 1-\max(0,\hat\tau_t)$, and clipped below by a ClipSMT-style floor $c_t$ so that every item keeps positive audit probability. Each audit updates the isotonic fit by inverse-probability-weighted pool-adjacent-violators and updates $\hat\tau_t$. The estimator above is reported together with an anytime-valid confidence sequence. That sequence is built from bounded martingale increments, which the clipping floor keeps bounded, using the empirical-Bernstein tools of Sharp Empirical Bernstein Inequalities.

The analysis has two parts. The first is a regret statement. When error is monotone in confidence and bounded away from 0 and 1 on a nontrivial fraction of items, the isotonic estimate of $e$ converges at the nonparametric rate $T^{-1/3}$. This rate is the reason ClipSMT's $t^{-1/3}$ clipping schedule is the natural match. We expect Neyman regret to grow sublinearly with an explicit dependence on the isotonic error. The second is robustness: under any monotone distortion of the confidence scale, including under- or over-confidence, RINA's allocation is unchanged, whereas plug-in rules are not. Two extensions are scoped but secondary. One audits for critical errors, as in Preventing Critical Scoring Errors, where $e_i$ is small and the Neyman gain should be largest. The other handles fallible auditors, the "perfect human" gap, through occasional double audits that estimate auditor noise as an extra variance term.

Baselines

The naive reference is uniform random auditing: every automated item is audited with probability $B/T$ and the plain sample mean is reported. This is current practice. Any adaptive design has to beat it at equal budget, or it has no reason to exist.

The second baseline is ClipSMT run over confidence strata. The automated stream is cut into ten confidence-quantile strata, and the per-stratum second-moment tracking of Logarithmic Neyman Regret is applied using its published clipping schedule. This is the strongest existing adaptive Neyman method. It is the right competitor because it already uses confidence, through the strata, and it differs from RINA only in not pooling across strata through the monotone fit. No reported result on Lacuna applies to audit estimation. The closest reported comparison is in Asymptotically Optimal and Computationally Efficient ATE Estimation: its $\mathcal{P}_2$ policy needed 2,672 samples against 4,034 for Clip-OGD in a Bernoulli A/B simulation, and it reduced sample size by about 50% relative to a randomized controlled trial in an exponential simulation. We include $\mathcal{P}_2$ over strata as well.

The third baseline is the calibrated plug-in rule: fit temperature scaling, or the class-wise signed variant from Two Sides of Miscalibration, on a held-out calibration set, then set $\pi_i\propto\sqrt{c_i(1-c_i)}$ from the calibrated confidence $c_i$.

Ablations isolate the key ingredients. Removing the isotonic step and fitting a logistic map from confidence to error tests whether the shape constraint, rather than pooling alone, matters. Removing the $\tau$ switch tests the safety mechanism. An oracle that uses true $e_i$ gives the variance floor against which Neyman regret is measured.

All methods see the same item streams, the same model confidences and the same audit budgets. The calibration-set baselines are given the labeled items their calibration needs, and those items are charged against the audit budget, because a deployment would have to pay for them. Hyperparameters such as stratum count and clipping constants get the same small grid for every method on a separate stream. RINA counts as beating a baseline if it reaches the same confidence-interval width with fewer audits, or a lower mean-squared error at equal budget, on most dataset–model pairs, while keeping nominal coverage. The target is at least a 20% budget saving over stratified ClipSMT in the miscalibrated settings and no loss in the well-calibrated ones.

Why not just…?

Why not just audit a uniform random sample? It is unbiased and simple, but it ignores that most automated items are almost never wrong. On the Complexity of A/B Testing proves uniform sampling is suboptimal under unequal variances, and in a delegated stream the variance is concentrated near the threshold. The experiments report the budget ratio between uniform auditing and RINA at matched interval width. If that ratio is near 1, for example because errors are spread evenly across confidence, the $\tau$ switch should detect it, and the honest conclusion would be that adaptivity does not pay.

Why not just calibrate the confidences once and plug them in? This is the most serious alternative. If the calibrated probabilities were accurate, the plug-in rule would be the Neyman-optimal design, and RINA would match it but not beat it. The case against it is twofold. Calibration fitted on held-out data drifts in deployment, as Balancing Cost and Quality observes for thresholds. And global temperature scaling can worsen under-confidence in Transformers, according to Two Sides of Miscalibration. We settle the question empirically by comparing the two on in-distribution streams, where we expect them to tie, and on shifted streams, such as a held-out prompt or an external cohort, where we expect the plug-in rule to lose. If the plug-in rule also wins under shift, the contribution reduces to the $\tau$ safety check, and we would report that result.

Why not just run ClipSMT over confidence strata? It is principled and already has guarantees. However, with ten strata and rare errors, each stratum's second moment is estimated from a few positives. Early allocations therefore stay near uniform for most of a realistic budget, and the low-confidence strata, which hold most of the variance, are estimated worst. Pooling through a monotone fit borrows strength across strata. The direct test is Neyman regret as a function of budget: RINA should pull ahead at small budgets, while the two methods should converge as the budget grows.

Why not just use the Trust Score or a better confidence measure? A better ranking helps RINA too, because RINA uses whatever score orders errors best. This is complementary rather than competing. We run every method with both posterior probability and the Trust Score on the short-answer-scoring data.

Proposed Experiments

All experiments replay existing labeled datasets as streams, so an "audit" reveals the stored gold label and the true error rate of the automated stream is known exactly. The short-answer setting uses ASAP-SAS and RIKEN with the delegation protocol of Balancing Cost and Quality. That paper reports 60–80% of grading automated at an RMSE tolerance of 0.16; we audit that automated portion for RMSE exceedance and for critical scoring errors, defined as in Preventing Critical Scoring Errors. The language-model setting uses TriviaQA and Natural Questions with Llama-2-chat confidences from NLL, following Uncertainty in Language Models: Assessment through Rank-Calibration, and the target is the correctness rate of answers the system emits without review. The miscalibration stress test uses ImageNet with a ViT, which is under-confident, and a ConvNeXt, which is over-confident, matching Two Sides of Miscalibration. Here the delegation threshold is fixed, and the confidence scale is additionally distorted by known monotone transforms.

The metrics are the mean-squared error of the error-rate estimate at budgets of 1%, 2%, 5% and 10% of the stream; the number of audits needed to reach an interval half-width of $\epsilon$; empirical coverage of the confidence sequence over 500 stream orderings; and Neyman regret against the true-$e_i$ oracle.

Three checks are decisive. The first applies monotone distortions to confidence. RINA's error curves should be essentially unchanged, while the plug-in baseline degrades. A failure here would falsify the central claim. The second shuffles confidences so that $\tau \approx 0$. RINA must fall back to uniform performance and not do worse, which tests whether the safety switch earns its place. The third is a shift experiment: calibrate on one prompt or question domain and stream another. This decides the contest between RINA and the plug-in rule. Coverage must stay at nominal level in every case. An efficiency gain bought with invalid intervals would count as a failure.