A deployed clinical risk model is useful only while acting on its predictions does more good than the trivial policies of treating every patient or treating none. Decision curve analysis measures exactly this. Its central quantity, net benefit, is the share of true positives minus a penalty on false positives, where the penalty is set by the risk threshold at which a clinician would intervene. Hospitals, however, check this quantity at most once, on historical data. As guidelines, coding practices, and patient populations change, a model can drift below the "treat all" line at some thresholds while still looking acceptable on AUROC.
We propose to give each deployed model a running, threshold-by-threshold lower bound on the net benefit it will achieve in the next deployment period. The bound has three properties. Its width adapts to how unstable that threshold has been. It self-corrects online when its past bounds turn out wrong. It still works when outcome labels arrive months late. The decision this bound supports is not whether to treat a given patient. It is whether the hospital should still use the model at a given threshold, or fall back to a default policy and retrain.
Core Claim
The obvious combination of these three directions is to wrap individual risk predictions in conformal intervals and draw "interval decision curves." We think that idea is dominated, and we drop it. Decision curve analysis assumes a risk-neutral decision-maker. Conformal Prediction and Human Decision Making shows that for such agents, calibrated probabilities strictly dominate "flat" conformal sets. Per-patient intervals would therefore add nothing to a treat/don't-treat rule that recent recalibration cannot already provide.
What recalibration cannot provide is knowledge of whether the model's policy will beat the defaults next period. That depends on drift that has not yet happened, and it is measured from labels that have not yet arrived. Our claim is that the right object to conformalize is the forecast error of net benefit itself, indexed by threshold and by time. Combining a per-threshold difficulty normalization with a lag-tolerant online correction should yield bounds that hold their target coverage across real clinical drift. They should also be tight enough to change deploy-or-fallback decisions compared with bootstrap confidence intervals or unnormalized bounds.
Prior Work and Where This Sits
The decision-theoretic grounding comes from two papers. A Consequentialist Critique of Binary Classification Evaluation Practices proves that net benefit integrated over a threshold window equals a bounded Brier score, and shows on breast-cancer screening data that model rankings can invert inside the clinical window. It works on static splits with point predictions only. Pfohl et al. show on 5.8 million patients that miscalibration makes net benefit collapse at a fixed guideline threshold of 7.5%. Their analysis compares subgroups cross-sectionally rather than following the same model through calendar time. Together these two papers motivate tracking net benefit over time, specifically near the thresholds clinicians use.
The temporal evidence both motivates and constrains the design. Guo et al. find that domain generalization and adaptation do not beat pooled training on MIMIC-IV, and that sepsis AUROC falls by up to 0.090 under the Sepsis-3 guideline change. A companion review by Guo et al. finds that recalibration repairs calibration drift but not discrimination decay. EMDOT supplies a rolling backtest protocol and shows that random splits are optimistic. It reports AUROC only. These results rule out a "make the model invariant" design and push us toward reactive maintenance, as argued by Feng et al.. They also leave open what we call Gap 3: once recalibration has fixed calibration, does lost discrimination still cap net benefit?
The interval machinery is reused from the conformal literature. Score normalization by a difficulty estimate (Boström et al., and the related quantile construction of Romano et al.) makes interval width track local difficulty. Adaptive conformal inference (ACI) (Gibbs and Candès) adjusts its error target online to keep long-run coverage under arbitrary drift, but it assumes each label arrives immediately. SPCI models autocorrelated residuals with quantile forests and has been evaluated on energy and finance series, never on clinical utility. We apply this machinery to a different target: the net-benefit trajectory of a model, rather than patient outcomes.
The closest work is Inverse Conformal Risk Control, which certifies upper bounds on regret and miscoverage across a family of uncertainty sets. It assumes exchangeable data, meaning past and future samples are statistically interchangeable. It also targets operations-research losses rather than the threshold-weighted clinical loss. We differ in three ways: the loss is net benefit, the data are non-exchangeable with delayed labels, and the output is a monitoring decision.
Research Program
Fix a model $f$, retrained or recalibrated on a schedule, and a grid of clinically relevant thresholds $p_t$. For deployment period $k$, the realized net benefit is
$$\mathrm{NB}_k(p_t) = \frac{\mathrm{TP}_k(p_t)}{N_k} - \frac{\mathrm{FP}_k(p_t)}{N_k}\cdot\frac{p_t}{1-p_t},$$
where $N_k$ is the cohort size in period $k$ and the counts come from treating everyone with $\hat p(x)\ge p_t$.
Before period $k$ begins, a simple forecaster produces a point forecast $\widehat{\mathrm{NB}}_k(p_t)$. Its default is the net benefit measured on the most recent labeled window; a smoothed trend is a variant. Once period $k$'s labels arrive, we compute a one-sided normalized score
$$s_k(p_t) = \frac{\widehat{\mathrm{NB}}_k(p_t) - \mathrm{NB}_k(p_t)}{\sigma_k(p_t)}.$$
The difficulty estimate $\sigma_k(p_t)$ combines two pieces: the binomial sampling standard error of net benefit at that threshold, which is large at thresholds where few patients fall near the cut, and the recent period-to-period volatility of net benefit at that threshold. Normalization is the adaptive-scaling ingredient. Without it, a stable high-volume threshold and a sparse, shock-prone one would share a single margin.
The certified bound for the next period is $\widehat{\mathrm{NB}}_k(p_t) - Q_{1-\alpha_k}\,\sigma_k(p_t)$. Here $Q$ is a quantile of past normalized scores, pooled across thresholds and recent periods. The level $\alpha_k$ is updated ACI-style, but only from periods whose labels have matured: when the period-$(k-d)$ outcome becomes observable at time $k$, it drives the update $\alpha_{k+1} = \alpha_k + \gamma(\alpha - \mathrm{err}_{k-d})$.
The hospital's policy at each threshold is to deploy the model only where the lower bound exceeds the better of "treat all" and "treat none." Otherwise it falls back to the default and flags the model for retraining. On the theory side, we will extend the Gibbs–Candès long-run coverage argument to a fixed delay $d$. We expect the coverage-error bound to grow roughly linearly in $d$, because each update reacts to an error that is $d$ periods stale. We will also handle right-censored outcomes by computing the counts with inverse-probability-of-censoring weights, following Pfohl et al.
A second, analysis-only strand answers Gap 3 directly. For each period we compare three curves: net benefit after sliding-window recalibration; net benefit after oracle recalibration on the period's own labels; and net benefit at a retrained model's discrimination. This splits the loss into a part recalibration could fix and a part only retraining can fix.
Baselines
The naive reference point is deploy-once. The model is trained on early data, its decision curve is computed once on a held-out historical split, and it is used for its whole decision window forever. It shows how much net benefit silently decays, which, as far as the file shows, no one has measured on these cohorts.
The strongest existing practice is pooled training with periodic recalibration, plus a bootstrap confidence interval on net benefit from the most recent labeled window. The model is deployed wherever the lower bootstrap limit beats both defaults. This combines the pooled-ERM model that Guo et al. found matched or beat every robustness method with the recalibration their review found effective. The bootstrap is what a careful clinical statistician would add. The only number the file reports for this pooled model is the 0.090 maximum sepsis AUROC drop together with one additional false negative per 11 septic patients for an outdated model. Net benefit for it was not reported in any work the file covers (no reported result found on Lacuna for net benefit on MIMIC-IV, SEER, OPTN or COVID cohorts). We will therefore compute this baseline's net-benefit trajectory ourselves.
Two ablations isolate the proposal's ingredients. The first is a fixed-width conformal bound: split conformal on unnormalized forecast errors, with no online update. It tests whether exchangeable conformal calibration already suffices. The second is unnormalized ACI with lagged updates. It tests whether the per-threshold difficulty scaling earns its place. An oracle that knows each period's realized net benefit provides the ceiling.
For fairness, every method uses the same base model, the same recalibration schedule, the same label delay, and the same threshold grid. The ACI step size and window lengths are tuned only on the earliest third of each timeline, with an equal grid budget for every method.
Beating the strongest baseline would mean two things over the remaining two-thirds of each timeline. First, our bounds hold within two points of the 90% target, while the bootstrap's realized coverage falls clearly below it, especially after documented shocks. Second, the deploy-or-fallback policy loses less cumulative net benefit relative to the oracle.
Why not just…?
Why not just recalibrate on a sliding window and redraw the decision curve? This is the most serious threat, and we include it as the main baseline rather than arguing past it. Recalibration does restore calibration (Guo et al.). But a redrawn curve describes the last window, not the next one, and it offers no guarantee near the narrow threshold windows that matter (Flores et al.). If recalibrated net benefit never actually drops below the defaults in our backtests, the bound is uninformative and we will report that. The Gap 3 decomposition will show directly whether that is the case.
Why not just put bootstrap confidence intervals on net benefit? Bootstrap intervals measure sampling noise inside a window that is assumed to be exchangeable. They say nothing about drift between that window and the next. We expect them to under-cover exactly at shocks such as Sepsis-3 or the SEER coding changes noted by EMDOT. Their coverage, broken down by pre-shock and post-shock periods, settles this.
Why not just monitor AUROC with a drift detector and retrain on alarm? AUROC can hold steady while calibration-driven net benefit collapses, as Pfohl et al. show. Conversely, AUROC can drop while recalibrated net benefit stays above the defaults. A detector on AUROC therefore triggers on the wrong signal. We will run an AUROC-alarm policy with a matched false-alarm rate and compare cumulative net-benefit regret.
Why not just make the model robust to drift up front? Guo et al. find that domain generalization and adaptation do not beat pooled training on MIMIC-IV. We therefore do not compete on that axis.
Proposed Experiments
We follow the EMDOT rolling backtest. The long timelines carry the online evaluation: SEER breast, colon and lung cancer (1975–2013, yearly periods), OPTN liver transplant (2005–2017), and the CDC and Southwestern Pennsylvania COVID-19 cohorts (2020–2022, monthly periods, where sliding windows were found to win). The SEER and OPTN survival outcomes give natural label delays and censoring. MIMIC-IV, using the four three-year windows and the four tasks of Guo et al., is too short for online calibration. We use it as a shock case study for the Sepsis-3 change, pooling scores across tasks and thresholds. Thresholds span each task's clinically plausible window; for comparison, Flores et al. use a breast-screening window of 1.66–3% and Pfohl et al. a statin threshold of 7.5%. Base models are logistic regression and gradient-boosted trees, since EMDOT found the two age differently.
The metrics are realized coverage of the lower bound, reported overall, per threshold, and around shocks. We also report the average gap between the bound and realized net benefit, and cumulative net-benefit regret of the deploy-or-fallback policy against the oracle. Finally, we report the fraction of each decision window certified useful.
Three outcomes are decisive. The first is that normalized lagged ACI keeps coverage near target where both the bootstrap and fixed-width conformal under-cover after shocks. The second is that removing normalization widens the bounds enough to fall back unnecessarily at stable thresholds, measurably increasing regret. The third is a delay sweep, $d$ from zero to several periods, showing how coverage degrades and whether it matches the predicted linear-in-delay behavior. If the bootstrap baseline already holds coverage under shocks, the conformal machinery is unnecessary and the contribution reduces to the Gap 3 measurement study, which stands on its own.
References
- Conformal Prediction and Human Decision Making
- A Consequentialist Critique of Binary Classification Evaluation Practices
- Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine
- Evaluating Model Performance in Medical Datasets Over Time
- Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare
- Accelerating difficulty estimation for conformal regression forests
- Conformalized Quantile Regression
- Adaptive Conformal Inference Under Distribution Shift
- Sequential Predictive Conformal Inference for Time Series
- Calibrating Decision Robustness via Inverse Conformal Risk Control