Teammate-Aware Clinical Alerts: Warning Only When the Care Team's Plan Falls Short

Hospital early-warning models look strong on historical records and then disappoint on the ward. A large share of the failure is not that they predict badly. They predict what the clinicians already know. A sepsis alert that fires two hours after the team has drawn blood cultures and started antibiotics is technically correct and practically useless, and enough such alerts teach clinicians to ignore all of them. Work on multi-agent coordination has already addressed a structurally similar problem. Robots and game-playing agents coordinate better when they share intentions, meaning their planned next actions, than when they share their current observations, because intentions stay valid while messages are delayed and they let each agent avoid duplicating what a partner is about to do. This proposal treats the care team as a teammate whose intentions can be read from the electronic health record (EHR) and forecast. The alert then fires on the residual: patients who are at high risk and for whom the team is unlikely to take a covering action within the warning horizon.

The second half of the claim concerns validation. Measuring the benefit requires an evaluation that looks like prospective deployment rather than a frozen retrospective table. We therefore replay records using the times at which data would actually have become available, not the times the events are labelled with. We expect intent signals, such as order placements, to hold up better under this replay than state signals such as lab results, because orders enter the record when a decision is made and results enter it later. This is the same asymmetry that makes intent sharing robust to message delay in multi-agent learning.

Core Claim

A clinical alert should be scored against a forecast of the care team's own next actions, not in isolation. Let $x_{\le t}$ be the patient data available by time $t$ and $I_t$ the team's observable intent at $t$ (active and recently placed orders, each weighted by its age). The alert score is

$$ s_t = P(y_{t:t+h}=1 \mid x_{\le t}, I_t)\,\bigl(1 - P(a_{t:t+h}\in \mathcal{C} \mid x_{\le t}, I_t)\bigr), $$

where $y_{t:t+h}$ indicates the adverse outcome within horizon $h$ and $a_{t:t+h}\in\mathcal{C}$ indicates that the team takes a covering action (for sepsis: cultures plus antibiotics, or a fluid bolus) within the same horizon. The second factor comes from a learned model of how the team behaves, and it plays the same role as the forecast of a teammate's future actions in multi-agent coordination. The claim has two parts, and each can be tested separately. First, at a fixed alert budget, $s_t$ produces far fewer redundant alerts than risk-only scoring and more alerts that actually come before the team acts. Second, under availability-time replay, the drop in performance is smaller for $s_t$ than for risk-only models.

Where This Sits

The mechanism comes from intent-based coordination. CoDe shows that agents exchanging learned multi-step behavioral intents, discounted by message age, keep a 68.6% win rate on SMAC under a 5-step communication delay. The state-sharing baselines TarMAC and NDQ fall to 43% and 38% in the same setting. We reuse CoDe's age discount to weight orders in $I_t$. ELIGN rewards agents for matching what their neighbors predict they will do. TAGET conditions offline policies on predicted teammate sub-goals and reports an average improvement of 37.83% over baselines. Both support the idea that a forecast of the partner's next move is the useful coordination signal. We do not train agents that talk to each other. Instead, we point a teammate-forecasting model at human clinicians who never change their behavior for the model, which makes this an ad hoc teamwork setting and not a communication protocol. SBRD predicts held-out sepsis-management actions in MIMIC-IV with 96.8% accuracy. That result makes it plausible that clinician actions can be forecast well enough to serve as the second factor. However, SBRD is an offline explanatory model and never uses its forecast to decide when to interrupt anyone.

The motivation comes from the validation literature. SepsisLab documents that commercial sepsis alerts fire after clinicians have already begun treatment. It responds with a single-agent 4-hour risk forecast and test recommendations, and it reports an AUC of 0.89 on MIMIC-III. It never models what the team intends to do, and it was assessed retrospectively together with clinician walkthroughs. The alert-fatigue problem that makes redundancy costly is described in Use of artificial intelligence in critical care and in the clinician survey in Assessing Barriers to Implementation. Mind the Performance Gap shapes the evaluation. In a prospective study of a C. difficile model, the authors attribute an AUROC loss of 0.016 to infrastructure shift, meaning differences between the research data warehouse and live operational feeds, while temporal shift slightly helped (+0.005). They also identify back-filled variables as a source of lookahead bias. The lesson is that honest offline evaluation has to reproduce when each data element becomes available. Yet Another ICU Benchmark (YAIB) constrains the design. It shows that cohort and label definitions move metrics more than model architecture does, so we fix YAIB's sepsis cohort and change only the scoring rule and the evaluation clock.

The closest multi-agent clinical work is The Optimization Paradox, which chains large language model (LLM) agents for information gathering, interpretation, and diagnosis. It finds that assembling the best individual models dropped diagnostic accuracy from 77.4% to 67.7%, with 13.87% hallucinated test results. MedAgentSim simulates doctor–patient dialogue among LLM agents. Both coordinate machine agents with one another inside a static case. Our proposal coordinates a single model with the real human team over live time, and it includes no LLMs.

Research Program

We will build three components on MIMIC-IV, using YAIB's sepsis cohort. The first is a risk model that forecasts the outcome over horizon $h$ from vitals, labs, and orders. The second is a teammate model trained by behavior cloning: it predicts whether a covering action occurs within $h$, using the same inputs. The third is the intent encoding $I_t$. Each order contributes a type embedding weighted by $\exp(-\lambda \Delta_t)$, where $\Delta_t$ is the time since the order was placed and $\lambda$ is learned. This is CoDe's timeliness discount applied to orders. We also include pending orders whose results have not yet been returned, because these are the clearest record of what the team currently intends to check.

One problem has to be designed around explicitly. The standard Sepsis-3 label in MIMIC depends on suspicion of infection, and suspicion is itself defined by cultures and antibiotics. That means the label is partly a record of clinician action. A model that "predicts sepsis" can therefore score well by predicting clinician recognition, which is exactly the redundancy we want to remove. We will report two outcome definitions. The first is the standard YAIB sepsis label. The second is an action-independent organ-dysfunction outcome, a SOFA increase of at least 2 (SOFA, the Sequential Organ Failure Assessment, is a standard organ-dysfunction score), defined without the suspicion criterion.

The evaluation clock is the second deliverable. Each data element is re-timed to the moment it would have become visible in the record. Labs are re-timed to their result or storage time, charted vitals to their entry time, and orders to their placement time. Variables that are known to be back-filled, such as weight and demographics updated after admission, are frozen at their first recorded value, following Ötles et al. We call this availability-time replay. We will release it as a transform over the YAIB pipeline. The central assumption is that MIMIC-IV's timestamps separate charted time from entry time and record when orders were placed. Before any modelling, we will audit coverage by field. Any intent source whose timestamps turn out to be unreliable will be dropped rather than imputed.

Baselines

The naive reference is a risk-only gradient-boosted tree model and a GRU (a standard recurrent network) trained on YAIB features. Alerts fire at a threshold chosen to produce a fixed number of alerts per 100 patient-days. This is the standard static-trigger design, and it tells us how many alerts are redundant when nothing is done about it.

The strongest existing method for early sepsis warning in the file is the SepsisLab approach: an LSTM that forecasts the 4-hour risk trajectory, with uncertainty estimated by Monte Carlo sampling. It reports an AUC of 0.89 on MIMIC-III (SepsisLab). We will reimplement it on the same MIMIC-IV YAIB cohort, because its reported number comes from a different database version. We will also strengthen it with the rule-based suppression that a hospital would plausibly add: no alert if antibiotics or cultures were ordered in the past six hours.

A third baseline gives the risk model the same order features that our method uses, but still scores with risk alone. This separates "orders are informative features" from "orders define what the team will cover".

For the question this proposal asks, the file contains no reported number for either redundant-alert fraction or lead time over clinician recognition. There is no reported result on Lacuna for these metrics on MIMIC-IV. The comparison therefore has to be generated fresh, with every method run under identical conditions. All methods share the same cohort, features, train, validation, and test patient splits, and hyperparameter search (the same number of Bayesian-optimization trials per method). All are compared at matched alert budgets of 1, 3, and 10 alerts per 100 patient-days.

Two ablations isolate the key ingredient. The first replaces the forecast teammate factor with the past-only suppression rule. The difference between that variant and the full method measures the value of anticipating team action rather than only reacting to actions already taken. The second removes the age discount, which tests whether CoDe's timeliness weighting matters when applied to orders.

Beating the strongest baseline means the following. At a matched alert budget, the full method must lower the fraction of alerts that fire after, or within 30 minutes before, the team's first covering action. It must do this without reducing the number of true outcome cases alerted with positive lead time. It must also show a smaller drop from charted-time to availability-time evaluation. Every comparison will carry patient-level bootstrap 95% confidence intervals, and the gains must hold for the action-independent outcome as well as the standard label.

Why not just…?

Why not just suppress alerts for patients who are already being treated? This is the most serious threat, and it is built in as the past-only ablation. Suppression can only remove alerts after the team has acted. It cannot hold back an alert in the hour before a covering action that the team was already about to take, and it cannot flag a patient whose partial workup (cultures drawn, no antibiotics) is heading toward delay. If the anticipation term adds nothing over suppression at matched budgets, the honest conclusion is that the contribution reduces to the replay protocol and the redundancy metrics. We will report it that way.

Why not just add orders as features to a better classifier? Orders are known to encode clinician suspicion, and that is exactly the problem. Including them raises AUROC by teaching the model to echo the team, which makes the model's alerts more redundant rather than less. The orders-as-features baseline tests this directly. The prediction is that its AUROC matches or exceeds our method while its redundant-alert fraction is worse. If both metrics favour it, the teammate factor is unnecessary.

Why not just use a stronger single forecaster or an LLM agent pipeline? Neither changes what is being scored. YAIB finds little gain from larger architectures on its ICU tasks. The Optimization Paradox shows that modular LLM systems can lose accuracy at handoffs, and it evaluated on static cases with no time dimension at all. A better risk model still fires on already-recognized patients. The residual score can wrap any risk model, so we will also apply it on top of the SepsisLab-style forecaster to show that the two approaches combine rather than compete.

Why not just average model and clinician judgments, as in reader studies? Score averaging between model and radiologist raised AUC to 0.891 in mammography (Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening). That setting has a single, simultaneous read. In the ICU there is no explicit clinician score to average at each hour, and the clinician's "judgment" is revealed only through orders over time. Forecasting those orders is the ICU analogue of the clinician's score.

Decisive Checks

Three results decide the project.

The first is the redundancy curve. We plot the fraction of alerts that fire after the team has already acted against the alert budget, for all methods, on both outcome definitions. The full method must beat suppression by a margin outside the confidence interval at budgets of 3 or more alerts per 100 patient-days.

The second is the replay gap. For each method we compute AUROC and alerts-with-lead-time under charted-time and availability-time clocks. The hypothesis is that order-based intent loses less than lab-based state, and it fails if the gaps are indistinguishable. As a sanity check we will report the magnitude of the gap next to the 0.016 infrastructure loss reported by Ötles et al..

The third is transfer. We retrain the teammate model on eICU and HiRID through YAIB and check whether it still anticipates covering actions at sites with different charting habits. Practice differs between hospitals, so we expect the behavior model to need local retraining. Measuring how much local data it needs is itself the useful answer for a prospective silent-mode pilot. The silent pilot is the natural next step, in which alerts are logged but not shown. It is only worth running if the first two checks pass.