Models that predict outcomes from intensive-care records usually start by chopping each patient's irregular stream of vitals and lab tests into fixed time windows, often one hour long. Inside each window they summarize what happened. Most summaries quietly record two different things at once: what the measurements showed, and how often something was measured. The second quantity is a strong predictor at the hospital where the model was trained, because clinicians check worried-about patients more often. It is also the quantity most likely to change at a different hospital, since charting frequency depends on staffing, device setup, and local protocol as much as on the patient. We propose to test a specific, falsifiable claim. A large share of the performance a window-aggregated ICU model loses when it is moved to a new hospital comes from the measurement-frequency part of its input, not the physiological-value part. If so, re-expressing frequency relative to each hospital's own charting norms should recover much of that loss. This re-expression needs only unlabeled records from the new site, not outcome labels.
The opportunity exists because two findings have never been put side by side. Deasy et al. aggregate 13,000+ raw MIMIC-III variables into hourly windows. They find that summing event embeddings, which preserves counts, beats averaging, reaching AUROC 0.878 for 48-hour in-hospital mortality. The authors themselves warn that this makes the model sensitive to staffing and charting intensity, and they never test another hospital. Yet Another ICU Benchmark (YAIB) runs systematic cross-dataset transfer across MIMIC-III, MIMIC-IV, eICU, HiRID and AUMCdb. It shows sharp drops when single-center models are moved, and it shows that preprocessing choices matter more than architecture. However, YAIB fixes a single hourly grid with forward-fill imputation and never asks which part of the aggregated signal carries the transfer penalty. We connect the two by treating the aggregation function as the object of external validation.
Core Claim
We make three claims. First, split each hourly window into a value channel and an explicit intensity channel. The value channel holds the latest or mean measurement per variable. The intensity channel holds the count of measurements per variable. Under this split, the cross-site drop in discrimination and calibration will concentrate in the intensity channel. Second, the size of that drop will track how differently two sites chart. HiRID's monitors record every two minutes, while eICU pools more than 200 hospitals with heterogeneous documentation, so the expected intensity mismatch between these datasets is large. Third, a site-relative intensity transform will keep most of the within-site benefit of counts while removing most of their cross-site penalty. The transform replaces raw count $c_{v,w}$ (number of measurements of variable $v$ in window $w$) with
$$\tilde{c}_{v,w} = \hat F_{s,v}(c_{v,w}),$$
where $\hat F_{s,v}$ is the empirical distribution of hourly counts of variable $v$ across all stays at site $s$, fit on unlabeled data. The model then sees "this patient is being checked more than is typical here," not "checked six times this hour."
This differs from every connecting paper in the file. MoRGen varies temporal resolution and finds coarse windows improve calibration, but it trains and tests within each dataset. It never transports a model, and it does not separate intensity from value. PathwayLLM does transfer a locked MIMIC-IV sepsis model to eICU, using a fixed 24-hour window. It reports no analysis of which input components fail. Cross-Representation Benchmarking compares binned and event-stream inputs, but only within MIMIC-IV and within EHRSHOT.
Research Program
We will build every model on YAIB's harmonized cohorts and task definitions: ICU mortality, sepsis onset and acute kidney injury. This keeps preprocessing identical across sites, which YAIB showed is the dominant confounder. We will extend the YAIB pipeline in one way: alongside its forward-filled value grid, each variable gets an explicit per-window count channel. Forward-filling otherwise erases counts.
Four input conditions share the same GRU and LightGBM learners and the same Bayesian tuning budget:
- Values only. Raw counts are removed.
- Values plus raw counts. This matches the spirit of Deasy's summation.
- Values plus site-relative counts. The transform $\hat F_{s,v}$ is fit on the target site's unlabeled stays.
- Counts only. This isolates how much outcome signal intensity carries by itself.
We will cross these conditions with window widths of 1, 6 and 24 hours. Coarser windows smooth intensity as well as values, and PathwayLLM's 24-hour choice may partly succeed for this reason.
For each ordered pair of the five datasets, and for eICU split by hospital, we will compute three quantities:
- Transfer penalty. Internal minus external AUROC and AUPRC, plus the change in expected calibration error.
- Intensity mismatch. The distance between the two sites' per-variable count distributions.
- Attribution. At the target site we permute the count channel across patients, then separately permute the value channel, and measure how much each permutation degrades external performance.
The main analysis asks whether intensity mismatch predicts transfer penalty across roughly 20 dataset pairs and many eICU hospitals. It also asks whether the site-relative transform flattens that relationship.
Baselines
The naive reference is YAIB's own pipeline: hourly binning, forward fill, and a tuned GRU or LightGBM trained on one dataset and applied frozen to another. YAIB reports internal mortality AUROC ranging from 0.84 to 0.87 across datasets. The file has no pair-specific transfer figure, so we will regenerate YAIB's transfer grid under our tuning budget rather than quote one.
The strongest intensity-exploiting competitor is Deasy-style summation aggregation, reimplemented on harmonized variables. It reports internal AUROC 0.878 at 48 hours on MIMIC-III, with no external result.
For a strong external reference on sepsis, PathwayLLM reports MIMIC-IV-to-eICU zero-shot AUROC 0.842, and 0.867 after light fine-tuning PathwayLLM. Its task definition and cohort differ from YAIB's, so we treat these as context, not a head-to-head target.
The strongest practical competitor is multi-center derivation. YAIB found eICU-trained models transfer better than MIMIC-trained ones. We will train each condition on eICU and test on HiRID and AUMCdb.
The key ablation is values-only versus values plus raw counts versus values plus site-relative counts, with learner and tuning held fixed. All conditions get the same trials, seeds and hardware. The site-relative transform must beat both alternatives on external AUROC and calibration while staying within noise of raw counts internally. We also report whether it adds to multi-center training rather than duplicating it.
Why not just…?
Why not just drop the counts? This is the real threat. If values-only models transfer as well as the site-relative version, the transform adds nothing and the correct recommendation is simply to discard intensity. Deasy et al. found counts carry signal internally, so dropping them likely costs something at the source site. Whether that signal survives transfer in any form is exactly what the ablation measures. We will report the result either way, including the case where the answer is "drop them."
Why not just recalibrate locally with a few labels? All models are local argues for recurrent local validation. Recalibration fixes miscalibrated probabilities, but it cannot fix a changed ranking of patients, and it requires labeled local outcomes. We will run Platt recalibration with small label budgets at each target site, both on its own and combined with our transform. This separates discrimination loss from calibration loss.
Why not just train on multi-center data? YAIB shows this helps. It still assumes the deployment site's charting falls inside the training mix. Pooling also often cannot happen, which is why MEDS-DEV evaluates in a decentralized way. We will test whether eICU-trained models still show an intensity-driven penalty on HiRID and AUMCdb.
Why not just use unbinned event streams? In Cross-Representation Benchmarking, CLMBR beats hourly binning within MIMIC-IV (AUROC 0.857 vs 0.806). Event streams encode intensity implicitly through token density, however, so they are not immune to this problem. Transfer of event-stream models is out of scope here. It is a natural follow-up using MEDS-DEV.
Decisive Checks
Two outcomes would falsify the thesis:
- The count-permutation test at the target site accounts for a small share of the transfer penalty.
- Intensity mismatch fails to predict the penalty across dataset and hospital pairs.
Two outcomes would support it but kill the method:
- Values-only models match site-relative models externally.
- Recalibration alone closes the same gap.
The method survives only under three conditions:
- Site-relative counts shrink the MIMIC-to-HiRID and MIMIC-to-AUMCdb penalties in both AUROC and calibration error.
- The shrinkage holds across the 1-, 6- and 24-hour windows.
- The gain persists on top of eICU multi-center training.
References
- Impact of novel aggregation methods for flexible, time-sensitive EHR prediction without variable selection or cleaning
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
- MoRGen: Mixture-of-Resolutions Generative Forecasting for Irregularly Sampled Medical Time-Series Data
- PathwayLLM: Explainable Clinical Trajectory Modeling with Structured Pathways for Sepsis Prediction
- Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction
- All models are local: time to replace external validation with recurrent local validation
- MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health