When a hospital deploys a triage model, it needs to know one thing quickly: of the patients the model flags, what fraction truly have the disease? This is the positive predictive value (PPV), and it decides whether clinicians trust the alerts or learn to ignore them. Retrospective numbers answer this badly. The deployed population differs from the training population, disease prevalence differs, and PPV depends heavily on prevalence. Confirming labels prospectively (chart review, follow-up, biopsy) is slow and expensive, and positives are rare, so a small audit gives a very wide interval. Two partial fixes exist. Reweighting the old labeled data to look like the new population uses importance weights, the ratio of target to source density for each patient. These weights explode where the new population is dense and the old one is sparse, and they are simply wrong when the relationship between features and disease shifts. Prediction-powered inference (PPI), which uses a model's predictions on a large unlabeled pool and corrects their bias with a small labeled sample, is valid only when the labeled and unlabeled samples come from the same population.
We propose a specific division of labor. Validity comes only from a small, deliberately designed audit of prospective target patients. We label flagged patients at known, bounded sampling probabilities. Efficiency comes from everything else: the large retrospective labeled set, reweighted toward the target, is used to build a better surrogate predictor, never to form the interval itself. Because density-ratio weights never enter the confidence bound, a bad weight estimate or a shift in the feature–disease relationship can widen the interval but cannot make it wrong. The question is whether this yields tight, valid PPV intervals from far fewer prospective labels than auditing alone. A second question is whether it stays valid in exactly the regimes where reweighting-based certification either fails or gives up.
Core Claim
Let $F$ be the set of target patients flagged by the deployed model. Let $Y\in\{0,1\}$ be the confirmed outcome and $g(X)$ a surrogate score estimating $P(Y=1\mid X)$ on the target. The target PPV is $\theta = \mathbb{E}_T[Y \mid X\in F]$. We estimate it as
$$\hat\theta = \frac{1}{N_F}\sum_{i=1}^{N_F}\lambda\, g(X_i) \;+\; \frac{1}{m}\sum_{j=1}^{m}\frac{Y_j - \lambda\, g(X_j)}{m\,\pi_j}\cdot \frac{m}{\hat N_F}.$$
The first sum runs over all $N_F$ flagged target patients, labeled or not. The second runs over the $m$ audited flagged patients, each sampled with a probability $\pi_j$ that we choose in advance. $\hat N_F$ is the design-weighted count of flagged patients, and $\lambda\in[0,1]$ is the power-tuning parameter of PPI++. The correction term uses design weights $1/\pi_j$, which we set ourselves and which are therefore bounded and exactly known. This is the design-based logic of DSL, applied to a clinical audit.
The source data enters only through $g$. We fit it on retrospective labels reweighted toward the target with stabilized density ratios and then recalibrate it, using cross-fitting so that no audit label touches $g$. Validity then holds for any $g$, under covariate shift, label shift, or concept drift. The quality of the reweighting affects only the residual variance $\mathrm{Var}(Y-\lambda g)$ on the target. For the bound itself we use an empirical-Bernstein construction on the bounded residuals rather than a normal approximation, because flagged positives in small audits are few.
Relation to Prior Work
Three papers motivate the gap. ShiftBench certifies per-cohort PPV under covariate shift with density-ratio weights, an empirical-Bernstein bound, and stability gates on tail index, effective sample size and clip mass. It never uses target labels or a rectifier. It assumes $P(Y\mid X)$ is invariant, and it certifies only 0.03% of molecular cohorts because the effective sample size collapses. LASS calibrates language-model surrogates for biomedical inference but explicitly leaves covariate shift open. Anytime-valid PPI assumes identically distributed streams and notes its coupling is fragile on skewed data.
Two papers constrain the design. ShiftBench proves that weighted conformal prediction is anti-conservative for PPV, with false certification rates of 65–81% at nominal 5%. That rules out routing validity through conformal PPI under shift. The latter paper's own "labelled-only paradox," where the audit alone beat prediction-powered methods, is why we keep $\lambda$ tuning and treat audit-only as a serious baseline. Double-weighting and the broader weight-variance literature (Sugiyama) show that stabilized weights help learning. We reuse them only for fitting $g$, which is the role they were shown to serve.
The method pieces come from PPI, PPI++, DSL, and ShiftBench's empirical-Bernstein machinery. The prospective literature supplies the target application. The Mia screening rollout reported PPV changes of 0.1–1.9% only as point rates. The ED sepsis trigger showed that retrospective AUC of 0.86 says nothing about live behavior. Our audit is the missing interval-producing step for deployments like these.
Research Program
The work has three parts.
First, we establish finite-sample validity for the design-weighted rectifier with an empirical-Bernstein bound. We characterize how interval width scales with the target residual variance. Width should shrink roughly as $\sqrt{\mathrm{Var}(Y-\lambda g)/m}$, following the standard variance-driven concentration form. We also derive the optimal audit design. Oversampling flagged cases near the decision boundary, where $g$ is least certain, should minimize width. This is the class-imbalance lever: labels are spent where rare positives and hard cases concentrate.
Second, we study surrogate construction. The candidates are an unweighted source fit, a clipped importance-weighted fit, a double-weighted fit, and a PSIS-stabilized fit. Each is measured by the target residual variance it produces, so that weight explosion becomes an efficiency cost we can quantify rather than a validity failure.
Third, we extend to sensitivity. Sensitivity is a ratio estimand that requires auditing unflagged patients, where positives are rarest and prediction power should matter most. We also give a sequential version for a prospective stream, using a nonasymptotic betting-style bound in place of the asymptotic coupling.
Baselines
The naive reference is the source PPV: the retrospective PPV reported at validation time, carried over unchanged. We expect it to be biased whenever prevalence differs. Beating it means correct coverage where it misses.
The first validity baseline is audit-only inference. It runs the same $m$ design-weighted labels through the same empirical-Bernstein bound with no predictions ($\lambda=0$). This is the strongest competitor whenever $g$ is weak. We beat it if we reach equal or narrower width at every tested $m$, and cut the labels needed to certify PPV ≥ τ by a meaningful margin. The PPI paper reports 316 versus 799 labels on AlphaFold as a reference for the scale of saving. We found no reported result on Lacuna for PPI-based PPV audits on clinical data.
The strongest existing shift method is the ShiftBench protocol run on source labels only, with the same density-ratio estimators, gates and Holm correction. ShiftBench reports zero false certifications across 19,600 boundary-null trials, and split conformal as the most powerful valid method at 16.46% of tabular cohorts certified. We beat it if we certify more cohorts at matched family-wise error under covariate shift. We also expect it to lose validity under label shift, where it is not designed to hold and our method is.
The "obvious combination" baseline puts importance weights directly into a PPI rectifier computed on source labels, with clipping. It gets the same data and weight estimators.
Two ablations isolate our key ingredients. One replaces the reweighted surrogate $g$ with an unweighted source fit. The other replaces the designed flagged-oversampling audit with uniform sampling. All methods share the same trained deployed classifier, the same audit sets per trial, and the same tuning budget for density-ratio models.
Why not just…?
Why not just label a target sample and skip predictions? This is the real threat. If $g$ explains little target variance, $\lambda$ tuning collapses our method to exactly this baseline, and the gain is zero. The experiments report label savings as a function of the surrogate's target PR-AUC. That tells practitioners when the extra machinery pays off. If the savings are negligible at realistic clinical PR-AUCs, such as the 17.24% that PubMedBERT reports for MIMIC-IV ED diagnosis, we will say so.
Why not just certify from reweighted retrospective labels, as ShiftBench does? It needs no prospective labels at all, which is attractive. But its guarantee assumes the feature–disease relationship is unchanged, while clinical deployment typically changes prevalence. Its own results show near-zero certification when effective sample size collapses. We test both failure modes directly by injecting prevalence shift into ShiftBench-style cohorts.
Why not just put importance weights in the PPI correction term? Validity then depends on correctly estimated weights and on covariate shift holding exactly. Its variance inherits the weight explosion documented by Sugiyama and by ShiftBench's gate failures. We run it as a baseline and expect undercoverage under label shift and wide intervals under severe covariate shift.
Why not use conformal PPI? ShiftBench's proof that weighted conformal bounds are biased upward for PPV argues against it on clinical metrics. We include it to confirm that anti-conservatism transfers to the prediction-powered setting.
Proposed Experiments
The first testbed is MIMIC-IV, following the COP-IV setup. The source is ICU admissions and earlier anchor-year groups. The target is ED visits and later years, standing in for a prospective deployment. The deployed model is a PubMedBERT classifier for a rare diagnosis. Full target labels are held out, so the true PPV is known and coverage is exact.
The second testbed is the 38-dataset ShiftBench suite. Each cohort serves as a target with simulated audit budgets of $m\in\{25,50,100,200\}$ flagged cases. We run two variants, one with the original covariate shift and one with additional injected prevalence shift.
The metrics are coverage of true target PPV at nominal 95%, interval width, and labels to certify PPV ≥ τ. We also measure the false certification rate on boundary-null trials, where true PPV equals τ, and the fraction of cohorts certified at family-wise error ≤ 0.05.
Three checks decide the outcome.
First, under injected label shift, our method must keep coverage near nominal while source-only certification and weighted-rectifier PPI undercover.
Second, the boundary-null audit must reproduce ShiftBench's standard of essentially zero false certifications.
Third, on MIMIC-IV the reweighted surrogate must certify with fewer audit labels than audit-only inference. The designed oversampling ablation should account for a measurable share of that gain. If the ablation without reweighting matches the full method, the importance-weighting component is unnecessary and we drop it.
References
- PPI++: Efficient Prediction-Powered Inference
- Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models
- ShiftBench: A Benchmark for Per-Cohort Certify-or-Abstain Decisions on Positive Predictive Value Under Covariate Shift
- Language Model Augmented Semi-Supervised Statistical Inference
- Anytime-valid, Bayes-assisted, Prediction-Powered Inference
- Extending Prediction-Powered Inference through Conformal Prediction
- A Unified View of Double-Weighting for Marginal Distribution Shift
- Supervised Learning under Covarita Shift
- Prediction-powered inference
- Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
- Revisiting Clinical Outcome Prediction for MIMIC-IV