Which Parts of an ICU Model Should Be Re-fit Locally? Separating Stable Patient Risk from Acute State

ICU risk models usually keep their ranking ability when moved to a new hospital, yet they can still give bad advice there. A model can sort patients correctly while its probabilities run too high or too low for the new population, and a cut-off chosen at the original hospital then triggers too many or too few interventions. The standard fix is to re-fit the model's probabilities on local data. That needs local outcome labels, which are scarce, and it treats every input the same way. This proposal asks a narrower question. Patient inputs fall into two kinds. Trait inputs are enduring characteristics such as age, chronic comorbidities, and pre-illness kidney function. State inputs are acute and depend on context: heart rate during resuscitation, ventilator settings, the worst blood gas of the day. State inputs are shaped heavily by local care protocols, so we expect them to be the main thing that drifts between hospitals. If so, a hospital should re-fit only the weight on the state part of the model and keep the trait part fixed. It should also choose its action threshold from the clinical cost of errors, not from the geometry of the ROC curve.

The test is direct. We train on MIMIC-IV, a single-center ICU database, and transport to the many separate hospitals in eICU. At each hospital we measure net benefit, the decision-analytic score that credits true positives and penalizes false positives at the exchange rate a clinician accepts. The core claim is falsifiable. Under transport, the calibration slope of the trait sub-score should stay near one while the state sub-score's slope drifts. Re-fitting only the state weight should then recover most of the net benefit of full local re-fitting, using fewer local labels. If trait weights drift as much as state weights, the claim fails, and we will report that.

Grounding and Gap

The state-trait distinction comes from psychiatry and neuroimaging. There, self-controlled longitudinal designs call a marker a state if it normalizes after recovery and a trait if it persists. Yin et al. use this design for depressive-episode connectivity. Seidel et al. find cerebellar elevations that persist after weight recovery in anorexia. Witt et al. separate mania-only from chronically dysregulated genes. These papers supply our method for labeling features. We borrow the self-controlled logic and apply it to MIMIC-IV patients with repeated ICU admissions.

Demirci and Calhoun motivate the transport question. They argue that confusing state with trait harms repeatability, and they report a classifier that reached 85% accuracy on 14 subjects but failed when scaled to more than 100 subjects across multiple sites. They never test this claim with a decision-theoretic metric.

The ICU transport work shows that discrimination can survive a move between databases. Qin et al. report weaning AUROC of 0.80 internally (MIMIC-IV) and 0.86 externally (eICU). Hou et al. report mortality AUC of 0.912 on eICU and 0.855 on MIMIC-III, against 0.741 for logistic regression. Neither paper reports net benefit. Both mix acute and baseline inputs in one model, so they cannot say what drifts. Ayaru et al. do the same for GI bleeding across two London hospitals.

The decision-analytic constraint comes from two papers. Pfohl et al. show that a fixed clinical threshold applied to miscalibrated probabilities destroys net benefit, so calibration is the precondition for utility. Flores et al. show that model rankings can flip once evaluation is restricted to a clinically plausible threshold window. They also prove that the Brier score restricted to that window equals integrated net benefit, and they release briertools. We reuse their bounded score as our primary metric. Their analysis stays within single datasets, and that is the setting we extend.

On thresholds, Liu et al. show that performance estimates at a cut-off fitted on the same sample are over-optimistic. They tighten inference at the cut-off that maximizes Youden's index (sensitivity plus specificity minus one, which weights both error types equally). They never ask whether that cut-off has positive net benefit at another site.

Wong et al. argue that "all models are local." They cite the Epic Sepsis Model, advertised at AUC 0.73–0.83, falling to 0.60–0.64 in independent evaluations. Their remedy is recurrent local re-fitting. They do not say which components need it, and our work is meant to fill exactly that gap.

Core Claim

We write the model as a sum of two sub-scores on the log-odds scale:

$$\text{logit}\, p(y=1\mid x) = a + b_T\, s_T(x_T) + b_S\, s_S(x_S),$$

Here $x_T$ and $x_S$ are the trait and state inputs, and $s_T$ and $s_S$ are sub-models trained on MIMIC-IV, each a gradient-boosted tree on its own feature block. The terms $a$, $b_T$, and $b_S$ are scalar recalibration parameters, equal to $(0,1,1)$ at the derivation site.

The claim has three parts. First, at external eICU hospitals the fitted $b_T$ stays close to one and varies little across hospitals, while $b_S$ and $a$ move. Second, fixing $b_T=1$ and re-fitting only $(a, b_S)$ from a small local sample recovers most of the net-benefit gain of full re-fitting. Third, choosing the action threshold at the clinically specified probability $p_t$ beats the transported Youden cut-off on net benefit within the clinical window.

The additive structure forbids state-by-trait interactions, which may cost some discrimination. We measure that cost directly and do not assume it away.

Research Program

Feature labeling without external data. In MIMIC-IV we take patients with two or more ICU admissions. For each candidate first-24-hour input we compute the intraclass correlation (ICC): the share of its variance that lies between patients rather than between one patient's admissions. This is the self-controlled design from the psychiatric literature, with readmission playing the role of a second episode. High-ICC inputs are labeled trait and low-ICC inputs state, at a threshold fixed before any eICU data are touched. Clinicians separately produce an a priori partition (pre-admission versus acute). We report agreement between the two partitions and run every analysis under both. Only derivation data are used, so the partition cannot leak information about the external sites.

Tasks. The primary task is ICU weaning success in sepsis, following the cohort definition of Qin et al.. It maps onto a real decision, whether to attempt liberation from the ventilator. The secondary task is in-hospital mortality as in Hou et al., run in the MIMIC-IV-to-eICU direction. For each task, clinicians pre-specify a threshold window of plausible $p_t$ values before evaluation. We report results inside that window and across a sensitivity sweep, because the file contains no agreed window for these decisions.

Transport protocol. Models, feature definitions, and the partition are frozen after derivation. Each eICU hospital with enough outcome events is a separate external site. At each site we draw local recalibration samples of increasing size, for example 25 to 400 labeled stays, and evaluate on that hospital's remaining stays. Hospitals, not patients, are the unit for uncertainty, estimated by bootstrap over hospitals.

Diagnostics. For each hospital we fit $(a, b_T, b_S)$ on all its data, using only to characterize drift and never for evaluation. We then relate the between-hospital variance of each parameter to hospital-level descriptors available in eICU. This tests the mechanism behind the claim, that protocol variation shows up through state inputs.

Baselines

The do-nothing references are treat-all and treat-none. Every strategy must beat both inside the clinical window to count as useful at all. The naive transport reference is the frozen MIMIC-IV model with its Youden cut-off, chosen on MIMIC-IV and carried unchanged to each eICU hospital. This is the practice that Liu et al. refine statistically but never test for utility. A second reference is the frozen model with the threshold set to $p_t$ itself, with no recalibration. It isolates how much of the gain comes from cost-aware thresholding alone.

The strongest competitor is standard whole-model logistic recalibration at each hospital. It fits an intercept and a single slope on the full log-odds from the same local sample, under the same label budget and the same evaluation split. It has two parameters against our two free ones, so it is the cleanest test of whether the state/trait split adds anything. We also run full re-fitting of all three parameters $(a,b_T,b_S)$. Recurrent local re-fitting in the spirit of Wong et al. means retraining the gradient-boosted model on local data. That is the upper reference at large label budgets.

For discrimination, the reported comparison points are AUROC 0.86 external for the full XGBoost weaning model of Qin et al. and AUC 0.855 external for the attention LSTM of Hou et al.. Our additive model is retrained on our cohort, so it is not directly comparable to these numbers. It should land near them, or we report the gap. No reported net-benefit or bounded-Brier result was found on Lacuna for either task, so the utility baselines are all run in this study under identical conditions.

The key ablation replaces the ICC-based partition with random partitions of matched block sizes. If re-fitting a random "state" block works just as well, the state/trait content is not doing the work. A second ablation re-fits only $b_T$, which our hypothesis predicts should help least.

Beating the strongest competitor means two things, both measured by the hospital-level paired bootstrap. First, higher integrated net benefit, measured as bounded Brier score via briertools, at small label budgets, roughly 100 local labels or fewer. Second, at least matching whole-model recalibration at larger budgets.

Why not just…?

Why not just recalibrate the whole model locally with an intercept and slope? This is the real threat. It costs the same number of parameters and is standard practice. It will match our method whenever state and trait sub-scores drift in the same proportion. Our method should win only if they drift differently, which is the empirical question the diagnostics answer. If the fitted $b_T$ and $b_S$ move together across eICU hospitals, the partition adds nothing. We would then report that the useful finding is the diagnostic, and that single-slope recalibration suffices.

Why not just pick the threshold by net benefit and skip recalibration? Pfohl et al. show that applying a fixed $p_t$ to miscalibrated probabilities degrades net benefit sharply. A cost-aware threshold is only as good as the probabilities it is applied to. The thresholding-only baseline measures exactly how much of the gap this leaves.

Why not just let a high-capacity model learn the distinction? The external AUROCs of Qin et al. and Hou et al. show that ranking survives transport. They say nothing about calibration or utility. The Epic Sepsis Model collapse reported by Wong et al. shows that capacity does not prevent learning site-specific artifacts. We still test the cost of our additive restriction against an unrestricted model trained on all features, and report it.

Why not just retrain locally, since all models are local? Retraining needs many local labels. The learning-curve comparison shows the label budget at which full local retraining overtakes partitioned recalibration. That crossover point is a practical answer to Wong et al.'s open question about what to keep global.

Decisive Checks

The hypothesis predicts a specific pattern across eICU hospitals. The variance of fitted $b_S$ should clearly exceed that of $b_T$ under both the ICC partition and the clinician partition. That gap should vanish under random partitions. Partitioned recalibration should beat whole-model recalibration on bounded Brier score within the clinical window at small label budgets. Finally, the transported Youden cut-off should show lower net benefit than the $p_t$ threshold, and at some hospitals fall below treat-all.

Two outcomes would refute the proposal. One is if $b_T$ drifts as much as $b_S$. The other is if the random-partition ablation performs equally well. Either result would still be informative, because it would show that the source of transport failure in ICU models is not the acute-state inputs, against the intuition voiced by Demirci and Calhoun.