Clinical early-warning models are usually judged as if every alert were read the moment it fires. The PhysioNet 2019 sepsis challenge scores alerts this way, and so do most AUROC comparisons. In a real ward, alerts pile up in a list that a few nurses or a rapid-response physician work through at a limited pace. Alerts also arrive in bursts. A stalled lab feed catches up all at once, or a busy admission hour produces many borderline scores together. An alert that sits unread for three hours has lost most of the value of predicting sepsis six hours early.
Communication networks faced the same problem decades ago. Sending traffic to the user with the best signal fails under bursty load, and the fix was to schedule using how much work is waiting in each queue. This note proposes to carry that idea into clinical AI in two ways. The first is an evaluation that replays historical ICU data through a simulated clinician review queue. The second is a small scheduling layer that sits on top of any risk model and decides which alerts to show, defer, or drop based on the current backlog.
Core Claim
The claim has two parts, and the first matters more.
First, rankings of alerting models built on the infinite-capacity assumption can reverse once a finite, bursty review queue is modeled. The finite-queue ranking should also better predict which model delivers value at a hospital held out from development. If this holds, part of the retrospective-prospective gap is an artifact of how models are scored, not only of drift in the data.
Second, a backlog-aware admission rule can recover a substantial share of the timely-alert value that capacity destroys. The rule comes from the drift-plus-penalty method of stochastic network control. It should beat both fixed alert thresholds and top-K alerting at the same total alert volume. The largest gains should appear during bursts caused by pipeline stalls.
The admission rule is simple enough to state. Let $Q(t)$ be the number of unreviewed alerts at hour $t$, $p_i(t)$ the model's risk for patient $i$, and $g_i(t)$ the drop in challenge-style utility if patient $i$'s alert is reviewed one hour later. Patient $i$ is queued only if
$$V\, p_i(t)\, g_i(t) > Q(t),$$
and the waiting alert with the largest $p_i g_i$ is served first. Here $V$ is a single knob trading expected utility against backlog. In Predictive Backpressure the analogous knob yields cost within $O(1/V)$ of optimal with backlog growing in $V$. We borrow the structure, not the guarantee. Clinician queues violate its assumptions, because service slows as the queue grows.
Relation to Prior Work
Two clinical papers motivate the gap directly. Reyna et al. defined a time-dependent utility that rewards alerts inside an actionable window and penalizes false and late alarms. Winning teams scored 0.33–0.36. The utility treats clinical response as having unlimited capacity, and we keep the utility while removing that assumption. Ötles et al. broke down a deployed infection model's prospective AUROC drop (0.778 to 0.767) into two parts. Infrastructure shift cost 0.016 and temporal shift gained 0.005, and the real-time pipeline silently failed on 10 of 356 days. We use their failure pattern as the template for injecting realistic stalls and catch-up bursts. They measured ranking and calibration, whereas we measure what those failures do to a review queue.
The closest existing work is Intervention Efficiency and Perturbation Validation. It scores models by true positives captured at a fixed capacity fraction $\gamma$. It also showed that capacity-aware selection picked the better-generalizing model 60% of the time at $\gamma=0.1$, versus 26.7% for single-split validation. Its capacity is a static share of a snapshot cohort, with no arrivals, waiting, or timeliness. Our evaluation replaces $\gamma$ with a queue that evolves hour by hour. Grand-Sloan et al. treated ventilator rationing as a sequential decision problem and found that official guidelines did no better than first-come-first-served. That result constrains us: first-come-first-served is a baseline we must beat, not a straw man. Their pool is beds, however, not a human review queue. ICU-Sepsis models single-patient treatment and assumes actions execute without contention, so it cannot express our problem.
On the networking side, Borst established that queue-length scheduling reaches capacity without any model of future arrivals. Huang et al. showed that under bursty arrivals, a strongest-signal heuristic met delay targets only 21.08% of the time, versus 70.71% for a queue-aware scheduler. That heuristic is the direct analogue of top-K risk alerting. Tavallaie et al. smoothed queue feedback with moving averages to stop oscillation. We adopt this because a flickering alert policy would be clinically unacceptable.
These wireless results supply the method, not evidence that it works in clinics. Human servers who tire as the queue grows are exactly the case the file flags as unstudied.
Research Program
We will build a queue-replay simulator. It takes hourly patient data, a trained risk model, and a clinician model, and replays each ICU unit's timeline. The clinician model has a number of reviewers, a distribution of review times, and a response probability that declines as the backlog grows, standing in for alert fatigue.
None of the datasets in the file record clinician review times. The clinician model is therefore a swept assumption, not a fitted one. Every headline result will be reported across a grid of reviewer counts, service rates, and fatigue slopes, and only conclusions that hold across the grid count.
Risk models come from the Yet Another ICU Benchmark pipelines. We train a deliberately varied set of models: linear, gradient-boosted, recurrent, and transformer. We then keep subsets whose AUROC agrees within a narrow band, reproducing the equal-accuracy "Rashomon" situation that Zhang et al. study. On top of each model we run the admission-and-ordering rule above, with the backlog smoothed by an exponential moving average. The model's inputs are never modified, which keeps the scheduling layer separable from the choice of risk model.
Baselines
The naive reference is the challenge's own practice: alert whenever the risk crosses a fixed threshold tuned for challenge utility, and review first-come-first-served. This is what a site would deploy after reading the retrospective results. Grand-Sloan et al. give reason to think first-come-first-served is not trivially beaten.
Top-K alerting per shift caps volume with a static risk ranking. It is the clinical twin of the strongest-signal heuristic that Huang et al. found weak under bursts. We run it at the same average alert count as our rule, so any win comes from timing rather than from sending fewer alerts.
The strongest competitor, and the ablation that isolates our key ingredient, is a backlog-adaptive threshold. The risk cutoff rises linearly with the smoothed queue length, and service stays first-come-first-served. It shares our feedback signal but ignores how quickly each alert loses value, so it tests whether the time-sensitivity term $g_i$ and max-weight ordering matter. A second ablation keeps our ordering but removes backlog-based admission.
All policies get the same risk scores, the same simulated clinicians, and the same tuning budget. We tune a single scalar per policy ($V$, threshold, K, or slope) by grid search on development hospitals, and never tune on the held-out hospital. For the reference point, the best reported challenge utility is 0.33–0.36 from Reyna et al., and this assumes unlimited review capacity. No reported result was found on Lacuna for sepsis alert utility under a finite review queue.
Beating the baselines means two things at matched alert volume. First, our rule must show higher queue-replay utility than the backlog-adaptive threshold across most of the clinician-parameter grid. Second, it must show a larger margin during injected stall-and-catch-up windows than in normal operation.
Why not just…?
Why not just raise the threshold or cap alerts per shift? Both control average volume but not timing. A burst from a lab feed catching up still arrives all at once, and a fixed cap cannot tell an alert that must be seen now from one that can wait. The strongest-signal result in Huang et al. suggests this matters under bursty load, but that evidence comes from radio scheduling. Our matched-volume comparison, reported separately for stall windows, is the direct clinical test.
Why not just make the threshold depend on the backlog? This is the real threat, and it may capture most of the gain. If it matches our rule within noise, we will report that. The contribution then narrows to the evaluation finding plus a recommendation for the simple rule, which would still be a useful and deployable result. The ablation is designed to settle this rather than argue around it.
Why not just build a more accurate model? YAIB found that preprocessing and cohort choices mattered more than architecture, with mortality AUROC clustered at 0.84–0.87. Ötles et al. traced the deployed drop to pipeline failures rather than model capacity. We test this directly: if the best-AUROC model also wins under queue replay at every capacity setting, the queue view adds little to model selection.
Proposed Experiments
The main testbed is the PhysioNet 2019 data. We develop on hospitals A and B and hold out hospital C, whose generalization drop the challenge already documented. Each YAIB dataset serves as an additional development-and-held-out split. Stalls are injected at the rate Ötles et al. observed. During a stall, lab-derived features freeze, and on recovery they are all released at once.
The first decisive check measures rank agreement between offline challenge utility and queue-replay utility across the equal-AUROC model sets, using Kendall's tau. It also asks which of the two rankings better predicts held-out-hospital queue utility. High agreement, or no predictive advantage, would falsify the first half of the claim.
The second check is the matched-volume policy comparison, including the backlog-adaptive ablation. It is broken out by normal hours versus stall windows and swept over the clinician grid.
The third check tests misspecification. We tune each policy under one clinician model and evaluate it under another, because a scheduler that only works when fatigue is modeled correctly would be unsafe to deploy.
A prospective silent-mode study is the natural next step. In it, the queue would be observed from real review timestamps rather than simulated. It is out of scope until the simulated results survive these checks.
References
- When backpressure meets predictive scheduling
- Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- Intervention Efficiency and Perturbation Validation Framework: Capacity-Aware and Robust Clinical Model Selection under the Rashomon Effect
- Interpretable Machine Learning for Resource Allocation with Application to Ventilator Triage
- ICU-Sepsis: A Benchmark MDP Built from Real Medical Data
- Inter-cell scheduling in wireless data networks
- Traffic-Aware Hierarchical Beam Selection for Cell-Free Massive MIMO
- GT-TSCH: Game-Theoretic Distributed TSCH Scheduler for Low-Power IoT Networks
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML