Alert Policies That Censor Their Own Evidence

A deployed clinical early-warning model does two things that retrospective validation never sees. It changes what clinicians do, and so it changes what data gets recorded. When a survival model (a model that predicts the full distribution of time until an event such as deterioration) flags a patient and a clinician intervenes, the deterioration it predicted may never happen. The untreated event time is then right-censored: we learn only that the event had not happened by the time of intervention. This censoring is not random. It falls on exactly the patients the model called high-risk, and it depends on whether a tired clinician, facing dozens of alerts, chose to act. Under naive monitoring the model therefore appears to overestimate risk after deployment, even when nothing about the patients has changed.

We propose to treat the alerting system as the leader in a Stackelberg game: a two-player setting where one side publicly commits to a strategy and the other responds to it. Here the leader commits to a randomized alerting policy under an attention budget. The follower is a boundedly rational clinician whose willingness to act on alerts decays with recent alert load and false-alarm rate. The randomization does two jobs. It is the optimal fractional allocation of scarce clinician attention, as coverage probabilities are in security games. It also supplies exactly known propensities, so the intervention-induced censoring can be undone with inverse-probability weighting. The core claim is that alert allocation and post-deployment calibration monitoring are one problem, and that solving them jointly beats top-$K$ thresholding on patient outcomes while removing a spurious component of the prospective calibration drift reported in live deployments.

Why this intersection, and what is new

Each direction has already gone halfway toward this problem. Survival calibration work supplies the predictive component. X-CAL turns distributional calibration into a differentiable training loss and tests it on MIMIC-III length of stay. Haider et al. define D-calibration: predicted survival probabilities at observed event times should be uniformly distributed. DeepSurv supplies a flexible risk model. All three treat the predicted curve as something passively consumed. All three also assume censoring is independent of the event given covariates, which is exactly the assumption a deployed alert breaks.

On the deployment side, Ötles et al. decomposed a prospective performance gap into infrastructure shift, which hurt discrimination, and temporal shift, which hurt calibration. Their decomposition does not separate true patient drift from the model's own effect on recorded outcomes. We take their finding as motivation and test whether part of the "temporal" calibration loss is actually performative censoring. Horng et al. reached AUC 0.86 for sepsis at emergency triage, but noted that retrospective design cannot say whether alerts produce bedside action. That is the follower-response gap we model.

The game-theoretic ingredients come from security games. Yang et al. showed that defenders optimizing against a quantal-response follower (BRQR) beat perfect-rationality solutions against human subjects. A quantal-response follower picks better options more often but makes noisy mistakes. The same paper found Prospect-Theory-based defenders to be brittle, and we treat that negative result as a constraint on design. Cerný et al. extend quantal Stackelberg solutions to sequential games. The closest applied work is Yan et al., who cast EMR audit-alert triage at Vanderbilt as a Stackelberg game with randomized prioritization under a budget. Their follower is a perfectly rational malicious insider, the setting is static, and no event times are involved.

Our follower is cooperative but fatigued. Our payoffs are survival trajectories. The leader's randomization is valued not for deterring anyone but for keeping the censoring mechanism known.

Core formulation

At each decision epoch, the model outputs an untreated survival curve $S_i(t)$ for each patient $i$ on a ward. It also outputs an expected benefit $\Delta_i$ if the clinician intervenes now. The leader chooses alert probabilities $x_i \in [0,1]$ with $\sum_i x_i \le B$, where $B$ is the alert budget per epoch. An alerted clinician acts with a logit (quantal) response probability $q_i = \sigma(\lambda(h)\,(\hat u_i - c))$. Here $\sigma$ is the logistic function, $\hat u_i$ is the clinician's perceived benefit from the displayed risk, and $c$ is the effort cost of acting. The term $\lambda(h)$ is a responsiveness parameter that falls as $h$, the recent alert volume and false-alarm fraction, rises. The leader's problem is

$$\max_{x}\; \mathbb{E}\Big[\sum_i x_i\, q_i(x)\, \Delta_i\Big] \quad \text{s.t.}\; \sum_i x_i \le B,$$

where $q_i$ depends on $x$ through $h$. This dependence is what makes the problem a genuine Stackelberg problem rather than a ranking: alerting more patients now lowers future $\lambda$.

When a clinician acts at time $a_i$, the untreated event time is censored at $a_i$. The probability of intervention, $\pi_i = x_i q_i + (1-x_i) q_i^{0}$, has a known leader part $x_i$. It also has two behavioral parts estimated from logged responses: $q_i$, the action rate given an alert, and $q_i^{0}$, the rate of unprompted action. Calibration of the untreated model is then monitored with an inverse-probability-of-censoring weighted D-calibration statistic. Each uncensored-by-intervention patient is weighted by $1/(1-\pi_i)$.

This weighting has a limit. Positivity, the requirement that $1-\pi_i$ be bounded away from zero, cannot be bought by withholding alerts from the sickest patients. We therefore restrict leader randomization to the marginal band near the budget cutoff, where the fractional optimum is randomized anyway. For the highest-risk strata we rely on the clinician's own dismissal rate, and we report explicitly which strata are identified and which are not.

Research Program

The work proceeds in three stages. Each stage produces a result that stands on its own.

The first stage is a semi-synthetic simulator built on MIMIC-III (Johnson et al.). It uses the intervention-onset tasks of MIMIC-Extract, vasopressor and ventilation onset, as deterioration events. We fit an MTLR or DeepSurv model with the X-CAL regularizer, and use a held-out "true" survival model to generate counterfactual untreated event times. Simulated clinicians follow fatigue-dependent quantal response with parameters we sweep. This lets us inject a known amount of real temporal drift, including none. The Stackelberg problem is solved by a sequential extension of the BRQR optimization. Where the horizon is short enough, we use the Dinkelbach-style reformulation of Cerný et al. We solve over a receding horizon because full extensive-form solutions do not scale.

The second stage is a human-subject triage study in the style of Yang et al. Nurses and residents work simulated shifts of patient timelines drawn from MIMIC-III, under different alert policies and alert volumes. The purpose is to estimate how $\lambda$ depends on load and false-alarm history, and to test quantal response against a Prospect-Theory account of how clinicians weigh displayed probabilities.

The third stage is a silent-mode prospective shadow deployment that uses the three-way data comparison of Ötles et al. That comparison sets retrospective warehouse data against live-extracted data and against a retrospective pull of the same window. In the shadow deployment, the policy's alerts and the recorded clinician actions are logged but not acted on differently from usual care. This lets us measure how much apparent calibration drift the censoring correction removes on real data before any randomized alerting is attempted.

Baselines

The naive reference point is no alerting: usual care, with the survival model used only for retrospective scoring. It sets the floor for the outcome metric. It also gives the calibration value obtained when no performative censoring occurs.

The standard practice baseline is deterministic top-$B$ alerting on the X-CAL-trained survival risk. Every patient above the budget cutoff is alerted. This is what most early-warning deployments do, and it isolates whether randomization and the game formulation add anything. For reference, the file reports mortality AUROC between 0.85 and 0.90 with AUPRC often below 0.50 for MIMIC-Extract models (Wang et al.). It also reports that X-CAL reduces D-calibration error "often by an order of magnitude" on MIMIC-III length of stay (Goldstein et al.). No reported result was found on Lacuna for any alert policy's effect on patient outcomes under clinician fatigue.

The strongest competitor is a behavior-aware but non-strategic policy. It fits a logistic acceptance model to logged clinician responses, multiplies each patient's benefit by predicted acceptance, and alerts the top-$B$ by that product. It captures bounded rationality but ignores how today's alerts change tomorrow's responsiveness. If fatigue feedback is weak, this baseline should match us, so it is the one we must beat.

We also run a perfectly rational follower Stackelberg policy, in which clinicians always act on alerts. This repeats the Yang et al. comparison in the clinical setting. A Prospect-Theory follower policy tests whether their negative result for Prospect Theory transfers. Two ablations isolate our ingredients. The first removes the dependence of $\lambda$ on history. The second rounds the randomized policy to deterministic, which shows how much identification is lost for monitoring.

For calibration monitoring, the baselines are naive D-calibration on deployment data and inverse-probability weighting with propensities estimated purely observationally, without known $x_i$.

All policies share the same survival model, patient streams, budget, and random seeds. Each gets the same hyperparameter search budget, tuned on a validation split of simulated shifts. Beating the strongest baseline means a statistically significant gain in the simulator in averted deterioration events per alert. The gain must hold across the fatigue-strength sweep whenever fatigue feedback is non-zero. It must be matched by the direction of effect in the human-subject study.

Why not just…?

Why not just alert the top-$B$ highest-risk patients? This policy ignores that clinicians respond to alert streams, not individual alerts. The evidence in the file that deterministic prioritization is exploitable (Yan et al.) concerns adversaries, so it does not transfer directly. The honest argument is different. Under deterministic alerting, the propensity of being alerted is 0 or 1, and the intervention-censoring of the untreated outcome becomes unidentifiable exactly near the cutoff. The rounding ablation measures this cost directly. If top-$B$ matches our outcomes and the calibration correction works equally well using clinician dismissal alone, the randomization claim fails.

Why not just fit an alert-acceptance model and threshold on benefit times acceptance? This is the real threat. If clinician responsiveness does not depend on recent alert load, the Stackelberg structure collapses to exactly this policy. Our human-subject study and shadow deployment test that dependence directly. If the estimated effect of load on $\lambda$ is indistinguishable from zero, we will report that the game adds nothing to the outcome side. We would then retain only the censoring-correction contribution.

Why not just retrain periodically on prospective data? Retraining on deployment outcomes teaches the model that alerted patients, who were treated and so did not deteriorate, were low risk. This degrades exactly the patients the system exists to catch. The simulator with zero injected drift shows this directly: naive retraining should lose discrimination on high-risk strata over successive rounds, while weighted retraining should not.

Why not just turn the problem into binary classification over a fixed window? Loh et al. show that this mislabels early-censored patients as negatives. Here, early censoring is caused by intervention on high-risk patients, which makes the label bias systematically adversarial to the model.

Decisive checks

The first check is the simulator with no injected temporal drift. Naive D-calibration on deployment data should show drift growing with alert uptake, while the weighted statistic stays flat. With injected drift, the weighted statistic should recover its true magnitude. The second check is the human-subject estimate of how responsiveness falls with load. It determines whether the Stackelberg policy has room to beat the acceptance-aware baseline, and it also tells us whether quantal response or Prospect Theory better predicts dismissals under fatigue. The third check is in the shadow deployment. We measure what fraction of the calibration drift, measured in the manner of Ötles et al., disappears after correcting for logged interventions. A near-zero fraction would mean performative censoring is not a practical contributor to the prospective gap in that setting, and we would say so.