Clinical early-warning models learn from two kinds of signal in a patient's record: the measured values (a lactate of 4.1) and the times at which measurements were taken (a lactate was ordered at all, at 2 a.m.). The second kind of signal is information sent by clinicians. A doctor orders a test when they are already worried, so the timestamp tells the model what the doctor suspects. Control theory has a name for this: implicit timing information. When a sensor transmits only after a threshold is crossed, the moment of transmission tells the receiver about the system's state even if the message carries no data. That same theory proves that delay corrupts this channel, and that once delay passes a critical value the advantage of timing disappears entirely (Time-triggering versus event-triggering control over communication channels). We propose to treat the timing of clinical measurements as such a channel, measure how much of a sepsis model's reported benefit flows through it, and test how that benefit holds up under the delays and ordering changes that exist in real deployment but are absent from retrospective benchmarks.
The point is this. Part of the clinical validation gap may come from a single measurable mechanism. Retrospective models partly read clinician suspicion back to clinicians, and they get credit for doing so. Deployment then breaks the channel they depend on. Lab results arrive after a turnaround delay rather than at order time. Hospitals order tests at different rates. Once alerts fire, the alerts themselves change ordering behavior. We will build a pre-deployment stress test that sweeps an injected delay on the timing channel and reports where utility collapses. We will also build a training method that keeps utility under realistic delay, and an evaluation that measures how far an alert precedes the clinician's own first workup, rather than how far it precedes a retrospectively assigned onset time.
Core Claim
Our hypothesis is that, on standard ICU sepsis benchmarks, a substantial share of model utility comes from measurement timing rather than measurement values. We expect this share to fall sharply once realistic result-availability delays and cross-hospital ordering differences are imposed, and we expect a large fraction of "early" alerts to come after clinicians have already begun a sepsis workup. If this holds, retrospective utility overstates deployable value in a predictable, measurable way. A delay sweep run before any silent trial would then expose which models are fragile.
The control-theory results supply the lens but not a theorem we can carry over. The Value of Timing Information in Event-Triggered Control and Exploiting Timing Information in Event-Triggered Stabilization of Linear Systems With Disturbances show for linear plants that the explicit bit rate needed for stabilization rises sharply with the delay bound $\gamma$. Time-triggering versus event-triggering control over communication channels locates a critical delay $\gamma = 1/h(A)$, where $h(A)$ is the plant's entropy rate, beyond which event timing no longer beats periodic sampling. Physiology is not a linear time-invariant plant, and clinicians are not deterministic triggers. We therefore borrow the question these papers ask, namely at what delay does the timing advantage vanish, and estimate the answer empirically rather than claiming the formula transfers.
Prior Work and Where This Sits
Two clinical papers motivate the proposal by showing that clinician behavior contaminates evaluation. Randomized Clinical Trials or Convenient Controls: TREWS or FALSE? fired purely random alerts across more than 650,000 encounters. Splitting patients by whether clinicians ordered lactate reproduced the claimed TREWS benefit: 3.3% versus 6.4% mortality in Cohort A, with the benefit vanishing under backdoor adjustment. That paper diagnoses confounding in the observational evaluation of alerts. We move the same logic upstream into model evaluation, using a model that sees only measurement timestamps as a learned analog of their coin flip. The PhysioNet 2019 challenge paper (Early Prediction of Sepsis from Clinical Data) introduced a utility metric that rewards alerts 1 to 6 hours before Sepsis-3 onset and showed degradation on a hidden third hospital. However, it scores lead time against an algorithmic onset label and treats each hourly record as available instantly. Our evaluation changes both assumptions.
Three modeling papers set the baselines and show the gap from the other side. All three treat time as a feature or a weight, never as a channel that can be corrupted. Development of a Real-Time Risk Prediction Model for In-Hospital Cardiac Arrest predicts time to arrest with a recurrent network plus a Weibull output, but scores it on a fixed grid with no response latency. EventScore builds a sparse point score whose training weights decay with time to event, and it was evaluated only retrospectively. Early Sepsis Prediction Using Ensemble Learning with Features Extracted from LSTM Recurrent Neural Network is the strongest documented challenge entry in the file.
Two papers constrain the design. Label-free estimation of clinically relevant performance metrics under distribution shifts found that every label-free monitor it tested fails systematically under prevalence shift. For that reason we do not rely on label-free monitoring to detect timing-channel breakage after deployment, and we test it in advance instead. A roadmap to implementing machine learning in healthcare prescribes silent trials. We regard our stress test as a cheap filter before them, not a replacement. In fairness to prior work, using missingness indicators as features is common in clinical machine learning. What is not in the material is a treatment of that signal as a delay-sensitive channel with a measurable breaking point, or scoring that signal against the clinician's own action time.
Research Program
The first component is a decomposition of each model's utility into timing and value contributions. For each model architecture we train three variants on identical data. The full variant sees values plus observation masks and time-since-last-measurement. The values-only variant sees forward-filled values with all timing features removed and replaced by a fixed hourly grid. The timing-only variant sees only the masks and intervals, with every value replaced by a constant. Writing $U$ for normalized challenge utility, we define the timing share as
$$ S = \frac{U_{\text{full}} - U_{\text{values}}}{U_{\text{full}} - U_{\text{null}}}, $$
where $U_{\text{null}}$ is the utility of never alerting. $S$ near zero means timing adds nothing beyond values. $S$ near one means the model's gain over silence is almost entirely in the timestamps. We also report $U_{\text{timing}}$ on its own, because a high score from a model blind to every lab value is direct evidence that the benchmark rewards reading clinician behavior.
The second component is the delay sweep. At test time we shift the moment each laboratory value and its mask become visible to the model by a delay drawn uniformly from $[0, \gamma]$, sweeping $\gamma$ from 0 to 6 hours. Vital signs are left unchanged as a control. This mirrors the bounded unknown delay in the control papers. We call the empirical critical delay $\gamma^*$: the smallest $\gamma$ at which $U_{\text{full}}(\gamma)$ falls to $U_{\text{values}}(\gamma)$, the point where timing stops paying. We will compare $\gamma^*$ against realistic turnaround. On MIMIC-IV, laboratory records carry both a sample time and a later storage time, which gives a real, per-test distribution of result-availability delay rather than an assumed one.
The third component is lead time over the clinician. We define the clinician-trigger time as the first lactate measurement or the first blood culture in the stay, both of which are standard sepsis workup actions. An alert counts as "ahead of the clinician" only if it fires before that time. We report the fraction of challenge-credited true positives that are ahead of the clinician, together with the median hours of lead over the clinician. A model can score well on challenge utility while almost never being ahead of the clinician; this evaluation is built to reveal that pattern.
The fourth component is a delay-robust training method: delay augmentation. During training we randomly delay lab visibility using the empirical turnaround distribution and randomly thin the observation masks to imitate a hospital with lower ordering intensity. The model is thereby forced to rest on values and on vitals, whose timing is protocol-driven rather than suspicion-driven. This is the component we expect to change practice if it works.
Baselines
The naive reference points are never alerting, which fixes $U_{\text{null}}$, and a rule-based score in the spirit of qSOFA computed from the available vitals. EventScore reports that data-driven scores beat MEWS and qSOFA, so we expect the rule-based score to lose on raw utility. It is still a meaningful reference: its inputs are sampled on a nursing schedule rather than by suspicion, so it should be nearly flat under the delay sweep. If our robust model does not beat it at realistic delay, the case for learned models weakens.
The strongest existing method is the LSTM-feature plus gradient-boosting ensemble of Early Sepsis Prediction Using Ensemble Learning. It reports normalized utility of 0.313 on the full hidden test set and 0.401 in internal 5-fold cross-validation, and the challenge paper reports winning teams between 0.33 and 0.36. We will reimplement it on the public hospitals A and B, using the same features, the same hyperparameter search budget as our model, and the same splits. Because the hidden Hospital C set may not be reproducible for us, the 0.313 figure serves as a sanity anchor and not as a direct comparison. We will also include a recurrent baseline with masks and time-since-last-measurement inputs, run under the same compute, since this is the conventional way of exploiting irregular sampling.
The key ablations are the values-only and timing-only variants described above. They isolate the timing channel itself. Delay augmentation with and without mask thinning isolates which part of the robustness comes from each corruption.
Beating the strongest baseline does not mean a higher zero-delay score. It means two things together: a higher $U_{\text{full}}$ at the median empirical turnaround delay, and a higher fraction of alerts ahead of the clinician on the held-out hospital. Both must hold, and we accept a small loss at $\gamma = 0$.
Why not just…?
Why not just remove the timing features and train a values-only model? This is the most serious alternative, and it may turn out to be nearly sufficient. If $S$ is small, or if the values-only model matches delay augmentation at realistic delay, then the honest conclusion is "drop the masks." In that case the contribution narrows to the diagnostic. However, there is a subtlety: forward-filled values still encode timing implicitly, because a value changes only when a new test is ordered. A values-only model therefore does not fully close the channel. The delay sweep applied to the values-only model will show whether that residual leak matters.
Why not just validate on an external hospital, as the challenge already did? Cross-hospital testing (Early Prediction of Sepsis from Clinical Data) shows that performance drops, but it mixes changes in patients, equipment, and ordering habits, and it says nothing about delay. Our A-to-B transfer experiment, with timing share measured in both directions, tests whether differences in ordering behavior explain a meaningful part of the external drop. The mask-thinning stress test then reproduces that part of the drop synthetically within a single hospital.
Why not just run a silent trial? Silent trials (A roadmap to implementing machine learning in healthcare) capture real latency, but they are costly and slow. They also cannot reveal the feedback effect, because alerts are suppressed and clinician ordering is unaffected. Observational readouts from active alerts are confounded in exactly the way the coin-flip study showed (TREWS or FALSE?). Our test is meant to decide which models deserve a silent trial. We do not claim it replaces one.
Why not just tune the alert threshold to manage false alarms? Moving the threshold changes how many alerts fire, but it cannot raise the share of alerts that come ahead of the clinician if the underlying score is driven by the clinician's own actions. We will show this directly by plotting the ahead-of-clinician fraction along the full threshold curve for each model variant.
Proposed Experiments
We will use the PhysioNet 2019 public training hospitals (Early Prediction of Sepsis from Clinical Data) for the main study, training on A and testing on B and vice versa. We will use a MIMIC-IV sepsis cohort, built with the MIMIC-IV processing pipeline, for real turnaround delays and clinician-trigger times. On MIMIC-III we will retrain an EventScore-style sparse score to test whether interpretable models leak timing as much as recurrent ones. The metrics are normalized challenge utility, area under the ROC curve, the timing share $S$, the empirical critical delay $\gamma^*$, the fraction of alerts ahead of the clinician, and the median lead over the clinician.
Three checks are decisive. The first is whether the timing-only model achieves utility well above zero; no reported result was found on Lacuna for a timing-only sepsis predictor. If its utility is essentially zero, the central premise fails and the proposal reduces to a negative result. The second is whether $\gamma^*$ for the strongest baseline falls below the median MIMIC-IV lab turnaround. If it does, the benchmark's reported advantage is unattainable in deployment as currently scored. The third is whether delay augmentation raises utility at realistic delay and raises the ahead-of-clinician fraction on the transfer hospital without a large loss at zero delay. A positive result on all three would show that a specific, cheap stress test predicts part of the clinical validation gap and that a matching training fix narrows it.
References
- Time-triggering versus event-triggering control over communication channels
- The Value of Timing Information in Event-Triggered Control
- Exploiting Timing Information in Event-Triggered Stabilization of Linear Systems With Disturbances
- Randomized Clinical Trials or Convenient Controls: TREWS or FALSE?
- Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019
- Development of a Real-Time Risk Prediction Model for In-Hospital Cardiac Arrest in Critically Ill Patients Using Deep Learning: Retrospective Study
- EventScore: An Automated Real-time Early Warning Score for Clinical Events
- Early Sepsis Prediction Using Ensemble Learning with Features Extracted from LSTM Recurrent Neural Network
- Label-free estimation of clinically relevant performance metrics under distribution shifts
- A roadmap to implementing machine learning in healthcare: from concept to practice
- An Extensive Data Processing Pipeline for MIMIC-IV