Residual Scores Make Sepsis Alarm Budgets Survive Hospital Shift

Bedside early-warning systems for rare, dangerous events such as sepsis face two problems at once. Positive cases are a tiny fraction of monitoring hours, and clinicians drown in false alarms. One attractive fix is an alarm rule with a statistical promise, for example "no more than 10% of the alarms we raise will be false". Such promises come from conformal prediction, a family of methods that turns a model's scores into guarantees using a held-out calibration set. The promise only holds if the patients being scored at deployment look statistically like the calibration patients. In ICU data they usually do not. Each patient has their own resting heart rate, vitals follow a daily (circadian) rhythm, and hospitals differ systematically. The PhysioNet/CinC 2019 sepsis challenge found that most models lost performance on an unseen hospital.

The proposal is to compute the alarm score on residuals rather than raw vitals. A residual here is what is left of each vital sign after subtracting that patient's own running baseline and a time-of-day component. The claim is that this subtraction makes the score distribution of non-septic patients nearly the same across patients, hours, and hospitals. If so, a false-alarm guarantee calibrated only on abundant non-septic data should keep holding when the system moves to a new hospital, and it should do so without spending scarce septic cases on calibration. The idea treats residual decomposition as a way of restoring the statistical interchangeability that guarantees depend on. Selective prediction (deciding when to act and when to stay silent) is the decision layer. Class imbalance is the constraint that rules out the usual class-stratified calibration.

Core Claim

Under severe imbalance, error-controlled alerting looks as if it is limited by the shortage of minority-class calibration data, but that is the wrong bottleneck. A rule that controls the false-discovery rate (the fraction of raised alarms that are false) needs calibration data only from negatives, and negatives are plentiful. The real limit is that negative-class scores computed from raw physiology are not exchangeable, meaning they cannot be treated as draws from one shared distribution, across patients, clock time, and sites. Residualization repairs this. We predict that a Benjamini–Hochberg alarm rule calibrated on residual-based scores keeps its realized false-discovery rate at or below target on a held-out hospital, while the same rule on raw-feature scores of equal or better AUC does not. We also predict that the residual version detects at least as many septic patients at matched realized false-discovery rate.

The closest work defines the gap. Act or Defer (STRATCP) wraps a frozen EHR foundation model with a class-specific Benjamini–Hochberg "action arm" and a prediction-set "deferral arm". On EHRSHOT pancreatic cancer at $\alpha=0.10$, it reports above 99% selected coverage for the positive class, where standard conformal prediction reaches only about 10%. STRATCP works on static code sequences, assumes calibration and deployment data are exchangeable, and explicitly lists distribution shift as an open limitation. We reuse its action-arm machinery and ask what score makes that assumption tenable in streaming physiology. COMPOSER abstains on ICU and ED windows that fall outside a conformal "trust set" built on raw recency-weighted features. It reports 5–10× fewer false alarms than deep baselines and misses only 1.1% of septic patients through rejection. Its rejection targets out-of-distribution inputs rather than a stated false-alarm budget, and it does not remove patient or circadian structure. Modular and Adaptive Conformal Prediction via Residual Decomposition decomposes prediction error across pipeline stages for regression intervals in finance. We decompose the input signal, for classification alarms under imbalance. Its sliding-window adaptation also needs fresh labels, which sepsis provides only with long delay.

The residual idea itself is borrowed from surveillance. STL for syndromic surveillance decomposes emergency-department counts into trend, yearly, weekly, and residual parts, and gains roughly 10% sensitivity over EARS and GLM. FlaSH removes weekday effects and changepoint regimes before an autoregressive residual test, reaching balanced accuracy 0.73 against DeepLog's 0.72. Both are population-level and offer no decision guarantee. We move the same logic down to the individual patient.

Research Program

For patient $i$, vital $v$, and hour $t$, we write

$$Y_{i,v,t} = B_{i,v,t} + C_{v}(h_t;\, \theta_i) + R_{i,v,t},$$

where $B_{i,v,t}$ is a causal, robust running baseline (a trailing median over the patient's own history up to $t$), $C_v(h_t;\theta_i)$ is a circadian curve over hour-of-day $h_t$ learned across the population with a patient-specific amplitude $\theta_i$ shrunk toward the population value, and $R_{i,v,t}$ is the residual. Labs are sparse and irregular, so they enter as residuals against the patient's last value with a recency weight, following COMPOSER's weighting idea. Every component is causal, so nothing peeks at the future.

A sepsis classifier $f$ is trained on residual windows. We use a gradient-boosted model and a small recurrent model to show the effect does not depend on architecture. Imbalance is handled with ordinary class weighting; the guarantee does not need $f$ to be calibrated.

The alarm layer computes, for each new window $x$, a conformal p-value against negative calibration windows:

$$p(x) = \frac{1 + \#\{j \in \mathcal{N} : s_j \ge s(x)\}}{1 + |\mathcal{N}|},$$

where $s(\cdot)$ is $f$'s septic score and $\mathcal{N}$ is a calibration set of non-septic windows. Benjamini–Hochberg is applied at level $\alpha$ over each batch of concurrent windows, and windows that are not selected are deferred.

Windows from the same patient are correlated, so $\mathcal{N}$ takes one randomly chosen window per calibration patient. We also report a patient-blocked variant. The analysis has two parts. The first measures exchangeability directly, as the two-sample distance between negative-score distributions across hospitals and hour-of-day bins, for raw versus residual scores. The second relates that distance to the size of the guarantee violations observed when transferring to a new hospital.

Baselines

The naive reference point is the trained raw-feature classifier with a single threshold, tuned on the source hospital to give a 10% false-alarm fraction and then applied unchanged at the target hospital. It shows how badly an untuned operating point drifts. Fairness comes from using identical features, the same training split, and the same threshold-selection data.

The strongest existing method is STRATCP's action arm (Act or Defer) applied to a raw-feature classifier of the same family, trained and tuned with the same hyperparameter budget as ours. It is the strongest competitor because it already delivers class-specific false-discovery control under imbalance. The reported number to beat is above 99% positive-class selected coverage on EHRSHOT pancreatic cancer at $\alpha=0.10$. There is no reported result on Lacuna for STRATCP on PhysioNet 2019 or any physiological stream, so we run it ourselves. Beating it means a realized target-hospital false-discovery rate at or below $\alpha$ where STRATCP-on-raw exceeds it, or equal validity with more septic patients alarmed before onset.

We also run a COMPOSER-style trust-set abstention (COMPOSER) on raw features. It is the strongest practical clinical comparator. Its reported reference numbers are AUC 0.925–0.953 in ICU cohorts and 1.1% of septic patients missed through rejection, both on its private cohorts. We compare alarms per patient-day and missed septic patients at matched realized precision.

The key ablation replaces the full decomposition with per-patient z-scoring against the first 24 hours plus hour-of-day as an extra feature. This isolates whether the structured subtraction matters or any patient normalization suffices. A second ablation removes only the circadian term.

Why not just…?

Why not just add patient baseline and hour-of-day as features to a raw model, then use STRATCP? This is the most serious threat. A flexible model can learn to compare vitals with their baseline, so its discrimination may match ours. The guarantee, however, depends on the score's marginal distribution among negatives at the new hospital, not on discrimination. A model can use baseline features well and still emit scores whose level shifts with a hospital's case mix. The ablation and exchangeability measurements settle this. If the feature-augmented raw model shows equal cross-hospital score stability and equal false-discovery validity, we will report that the benefit comes from patient normalization in any form, and the decomposition machinery is unnecessary.

Why not just use class-conditional (Mondrian) conformal prediction on raw scores? Stratifying by class addresses the marginal-versus-minority failure that STRATCP documented, and our method keeps that property. Stratification does nothing about patients, clock time, or hospitals, which are the axes the PhysioNet challenge identified as the generalization barrier (PhysioNet 2019). The STRATCP-on-raw baseline tests exactly this.

Why not just rebalance training with oversampling or focal loss and retune the threshold? Rebalancing changes discrimination and calibration, and Calibrated Selective Classification shows that such training-time changes distort probabilities. Retuning a threshold at a new hospital also needs labeled septic cases from that hospital, which is the scarce resource. We include a focal-loss raw model inside the STRATCP baseline to check that its gains are not merely a calibration effect.

Why not just adapt online with a sliding-window conformal method? Methods such as the adaptive scheme in Modular and Adaptive CP need recent labels, and sepsis labels arrive hours to days late. Residualization asks only for each patient's own unlabeled history. We will run a label-delayed sliding-window variant as a secondary comparison and report honestly if it closes the gap.

Proposed Experiments

The primary benchmark is PhysioNet/CinC 2019: hourly records of 40 variables, over 60,000 ICU patients, Sepsis-3 labels. The main test trains and calibrates on Hospital A and deploys on Hospital B, then reverses the direction. Hospital C is used if the hidden set is accessible.

For each method and for $\alpha \in \{0.05, 0.10, 0.20\}$, we report the following. Validity is the realized false-discovery rate among alarms at the target hospital, with bootstrap intervals over patients. Power is the fraction of septic patients alarmed at least 6 hours before onset. Burden is alarms per patient-day, together with median lead time. Finally, we report the challenge utility score, for which winning teams reported 0.33–0.36 (PhysioNet 2019), so that discrimination is not sacrificed unnoticed.

The decisive check has three parts. First, the residual score must show smaller cross-hospital and cross-hour divergence among negatives than the raw score. Second, it must be the only variant whose realized false-discovery rate stays below $\alpha$ at the target hospital. Third, the residual score must not lose discrimination AUC to the raw model; if it did, any validity gain could just reflect a more conservative rule. If the raw and residual scores transfer equally well, the core claim is false and we will say so.

A second check runs stress tests. We inject circadian amplitude changes and patient-level baseline offsets into the source-hospital negatives. A residual score should be nearly invariant to these perturbations, and a raw score should not.

A small replication applies the same alarm layer to EHRSHOT using STRATCP's tasks. This confirms our implementation reproduces the reported positive-class coverage on its native setting before we rely on it in the new one.