Training on What the Monitor Actually Knew: Arrival-Time Window Features over a Common Data Model

Most clinical early-warning models are trained on retrospective tables that place every measurement at the moment it was taken. At the bedside, a lab value taken at 2:00 may not reach the system until 3:30. A model trained to predict sepsis at 3:00 using the 2:00 lactate is therefore trained on information it will never have when it is deployed. This mismatch between training-time and serving-time features is invisible in offline benchmarks, because the window statistics are computed after the fact. We propose to measure it and remove it. The core artifact is one feature specification written against a common data model, meaning a shared schema and vocabulary into which every hospital's records are translated. The specification defines rolling-window summaries of standardized clinical concepts "as of arrival time." It runs in two modes that must produce identical outputs: an offline mode for training and an incremental streaming mode for serving. On top of that feature layer we test whether one model shared across several prediction tasks can serve many endpoints from the same windows without losing accuracy to per-task models.

In two sentences: today's window-feature pipelines quietly use measurements before they have arrived, so their reported accuracy overstates what a live system can achieve. We build a portable window-feature definition that only sees what has arrived, show the size of the overstatement across five ICU databases, and test whether training on honest features, optionally with a shared multi-task model, recovers the lost accuracy.

Core Claim

Our central empirical claim is that the standard practice of computing sliding-window statistics by measurement time ("event time") inflates offline performance relative to a faithful replay of the data stream. The inflation should be largest for sparse, slow-to-report concepts such as labs, and largest for short prediction horizons. Training on arrival-time features should close much of the gap between offline and live accuracy. Our methodological claim is that defining these features on harmonized concepts, rather than on site-specific columns, lets one specification run unchanged across sites, offline and online. This turns the train/serve consistency question into a testable property: bit-identical feature vectors from both executors.

This differs from every closest neighbor in the file. Strong window-feature pipelines such as the sepsis XGBoost system of Early Prediction of Sepsis Using Multi-Feature Fusion Based XGBoost Learning and Bayesian Optimization compute 6-hour min/max/mean/std and missingness counts offline over hourly records. They never ask when each value became available. The harmonization benchmarks Yet Another ICU Benchmark and MEDS-torch standardize concepts across sites, but they are batch-only. The multi-task foundation models EHRMamba and CEHR-XGPT learn from tokenized discrete events in static FHIR or OMOP stores and have no notion of a live stream. Streaming systems such as N-BaIoT and the aggregator of General Incremental Sliding-Window Aggregation maintain windows incrementally, but over raw tuples that arrive in order. No existing work treats arrival order as a property of the clinical data that the features must respect.

Research Program

Feature specification. The first component is a declarative window specification over harmonized concepts, using the ricu concept layer from YAIB. Each feature names a concept (for example, lactate or mean arterial pressure), a window length drawn from a multi-scale set (1 h, 6 h, 24 h, in the spirit of the nested lookbacks in Treatment Discontinuation Prediction in Patients With Diabetes Using a Ranking Model), and an aggregate. The aggregates are min, max, mean, standard deviation, count, time-since-last, and percentiles where cheap. Every feature value at query time $q$ is defined over the set of measurements with measurement time in $[q-W, q]$ and arrival time at or before $q$, where $W$ is the window length. Event-time features drop the second condition. That single clause is the object of study.

Data. MIMIC-IV records both when a lab or chart value was taken and when it was stored, so the arrival process can be observed directly there. For the other YAIB databases (MIMIC-III, eICU, HiRID, AUMCdb), we will check whether comparable storage timestamps exist. Where they do not, we will impose per-concept delay distributions estimated from MIMIC-IV. This is a modeling assumption, and we will report sensitivity to it rather than present it as ground truth.

Two executors. The second component is two executors for the same specification. The offline executor is a straightforward as-of query over the harmonized tables. The streaming executor builds on the tree-structured aggregator of General Incremental Sliding-Window Aggregation, which maintains associative aggregates such as Max and StdDev in a flat binary tree. That method assumes first-in-first-out arrival, and the file flags out-of-order arrival as an open problem for streaming aggregators. We address it by bucketing measurement time into fixed bins: a late value updates its bin's partial aggregate and the logarithmic path above it, rather than forcing a rollback. Correctness is defined as exact agreement with the offline executor on replayed streams.

Models. The third component is the modeling study. We train per-task gradient-boosted trees and a shared multi-task network on identical feature tables, under both event-time and arrival-time featurization. The tasks are YAIB's mortality, acute kidney injury, sepsis onset and length of stay, plus a short-horizon hypotension task in the style of A dual boundary classifier for predicting acute hypotensive episodes in critical care. For the multi-task network we follow the finding of A Comprehensive Evaluation of Multi-task Learning and Multi-task Pre-training on EHR Time-series Data: joint training suffered negative transfer, while multi-task pre-training followed by per-task fine-tuning avoided it and gave up to roughly 25% AUROC gains with 1% of the data. That paper constrains our design, so pre-train-then-fine-tune is our default. We include a joint-training variant with gradient balancing from RotoGrad only as a secondary arm.

What is not claimed. Multi-task sharing is a hypothesis, not the headline. If per-task trees win on arrival-time features, the headline claim still stands.

Baselines

The naive reference point is the current standard practice: event-time window features fed to per-task models and evaluated offline. Its reported numbers are the ones a reader would otherwise believe. Our evaluation re-scores the same models under stream replay, so this baseline also yields the size of the overstatement.

The strongest competitor is a per-task gradient-boosted tree on hand-built window summaries. Two results support this. Early Prediction of Sepsis Using Multi-Feature Fusion Based XGBoost Learning and Bayesian Optimization reports a local AUROC of 0.847 and a hidden-test Utility Score of 0.364 on the PhysioNet 2019 sepsis challenge. Yet Another ICU Benchmark reports LightGBM close to GRU and Transformer models, with mortality AUROC of 0.84 to 0.87 across its datasets. We run LightGBM per task on exactly our feature tables, with the same Bayesian-optimization tuning budget per task as the multi-task network receives in total. A multi-task model beats it only if it matches per-task tree AUROC and AUPRC within confidence intervals on all tasks while serving them from one model, or exceeds it on the low-data sites.

The pre-train-then-fine-tune multi-task network of A Comprehensive Evaluation of Multi-task Learning and Multi-task Pre-training on EHR Time-series Data is the strongest neural competitor. That paper reports its gains relative to single-task baselines rather than absolute AUROCs, so no reported absolute number on our tasks was found on Lacuna. We reimplement it on our features with the same encoder size.

For the streaming executor, the baseline is naive full-window recomputation. For windows up to roughly 10 to 370 elements depending on the function, General Incremental Sliding-Window Aggregation found recomputation was actually cheaper, with up to a 10x incremental advantage at window size 1,024. The key ablation is identical models with only the arrival-time clause toggled. This isolates the proposal's central ingredient.

Why not just…?

Why not just use per-task gradient-boosted trees, as the strongest window-feature systems already do? This is a genuine threat on accuracy, and the proposal is designed so that it does not need to beat trees to make its main point. The arrival-time correction applies to trees as much as to neural models. If trees trained on arrival-time features remain best, we report that as the recommended deployment. The multi-task question is settled by the head-to-head described in Baselines, especially on HiRID and AUMCdb subsampled to small training sets. Those small-data settings are where A Comprehensive Evaluation of Multi-task Learning and Multi-task Pre-training on EHR Time-series Data saw shared pre-training help most.

Why not just shift every feature back by a fixed reporting delay? A constant lag is the cheapest correction and could plausibly capture most of the effect. We therefore include it as a baseline: lag each concept family by its median observed delay. Delays in a hospital differ by concept, time of day and site, so we expect a fixed lag to over-penalize fast vitals and under-penalize slow labs. The replay evaluation will show directly whether the per-measurement arrival clause beats the best fixed lag. If it does not, we will say so and recommend the lag.

Why not just do offline as-of joins and skip the streaming engine? For the scientific claim this is correct: offline as-of queries suffice to train and evaluate honestly, and they are our reference executor. The streaming executor matters only for serving, where the risk is a second, hand-written implementation drifting from the training code. Its contribution is measured separately, as exact agreement with the reference plus latency and memory at realistic and high telemetry rates. We will also report the window sizes below which naive recomputation is the better engineering choice.

Why not just use an EHR foundation model like EHRMamba or CEHR-XGPT? Those models consume discrete event tokens from static stores. They do not define what is known at a query time inside an hour, so they inherit the same event-time leakage unless their inputs are rebuilt. A foundation-model comparison is out of our compute scope. Our arrival-time tables could, however, be emitted in event-stream form for MEDS-based pipelines as follow-up work.

Proposed Experiments

Main experiment: the overstatement gap. We train each model on event-time features and evaluate it two ways on held-out patients. The first is the usual offline evaluation. The second is a replay evaluation, in which records stream in arrival order through the streaming executor and predictions are made hourly from what has arrived. The gap between the two, in AUROC, AUPRC and, for sepsis, the PhysioNet utility score, is the headline quantity. It is reported per task, per prediction horizon and per concept family.

Recovery. We then retrain on arrival-time features and report how much of the gap closes under replay. The prediction is a nontrivial gap for sepsis and acute kidney injury, which lean on labs, and a small gap for vitals-driven hypotension. If the gap is negligible everywhere on MIMIC-IV's observed delays, the main claim is falsified, and we will publish that as a reassuring negative result.

Portability across sites. We train on one YAIB site and test under replay on the others, reusing the specification unchanged. This asks whether arrival-time training changes the cross-site degradation documented in Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019. Results on sites with simulated delays are labeled as such and swept over delay scale.

Multi-task versus per-task. We compare per-task trees, per-task networks, multi-task pre-train-then-fine-tune, and joint training with gradient balancing, at full data and at 1%, 10% and 100% subsamples, on arrival-time features only.

Systems check. On replayed MIMIC-IV streams, and on synthetic streams scaled up to waveform-like rates, we require bit-exact agreement between the two executors. We report per-update latency and memory against naive recomputation across window sizes.