When a Risk Model's Alerts Reach the Neighbors: Interference-Aware Prospective Validation

A hospital risk model for a contagious infection is usually validated prospectively as if each patient were an island. The model scores live encounters, clinicians act on its alerts, and accuracy and benefit are measured patient by patient. Infection and staff time do not respect that assumption. Suppose an alert sends one patient to an isolation room or pulls a nurse to their bedside. That action changes the risk and the recorded outcome of the patient in the next bed, and of everyone else that nurse covers. The labels that prospective validation relies on are then partly produced by the model's own alerts. Patient-level randomization of alerts also leaks benefit from treated patients to controls.

We propose a prospective validation design that treats the ward as an interference network, borrowing the main modeling move from wireless resource allocation. In wireless networks, each link's signal is noise to its neighbors, and graph neural networks (models that update each node by combining messages from its neighbors) learn how strongly each link interferes with each other link. We would learn an analogous patient-to-patient interference graph from historical admission, transfer, and staffing records. That graph then drives a randomized rollout of model alerts. The rollout is chosen so that direct benefit, spillover benefit, and ordinary accuracy degradation can each be estimated without bias. The two-sentence version is this: deploying a clinical model is an intervention whose effects spread across patients, so its prospective validation should be designed like an experiment on a network. We supply the network by learning it, wireless-style, from retrospective data.

Core Claim

The claim has three parts.

The first is that prospective validation of alert-driven models for transmissible or resource-limited conditions is biased in a specific, measurable way. Neighbors' alerts change both the outcome rate and the measured discrimination of the model.

The second is that the Conflict Graph Design can remove that bias. This design randomizes treatment while accounting for which units' outcomes cannot be observed cleanly at the same time. Its variance bound scales as $O(\lambda(H)/n)$, where $H$ is the conflict graph, $\lambda(H)$ is its largest eigenvalue, and $n$ is the number of patients. The design only works, though, if $H$ is close to the true interference structure. The administrative graph a hospital records, meaning who shared a room or a nurse, is not that structure.

The third is that we can learn $H$ from retrospective data with an edge-weighted message-passing model. The model is built in the style of wireless interference graphs and is checked against simpler graphs. Our contribution is the combination of learned graph, design, and gap accounting. None of these three exists as a deployable protocol in the material we found.

Where This Sits in Prior Work

The wireless literature supplies the modeling template. WCGCN treats each transmitter–receiver pair as a node and each cross-channel gain as an edge feature. It shows that a model trained on 50 pairs runs on 1,000 pairs with about a 1–2% loss, because the same weights are shared across nodes. REGNN and UWMMSE also transfer across network sizes. Map2Schedule predicts the interference weights themselves, from city layouts instead of measured channels. It shows that raw distance is a poor proxy: distance-based scheduling reached about 50% of potential capacity in urban settings, where buildings break the link between distance and attenuation.

Our analog maps floor plans, room assignments, staffing rosters, and transfer timing to edge weights. Size transfer matters here because ward census varies daily. The wireless papers assume the graph is known or computable from physics. We do not. They also optimize sum-rate rather than estimate causal effects.

The causal-inference papers motivate the problem and supply the estimation machinery. CauGramer shows that treating an observed social graph as the true interference graph badly biases effect estimates. Its attention-based peer model cut errors by 28–80% on semi-synthetic BlogCatalog and Flickr benchmarks. It works only on observational data and was never applied to clinical topology or live streams. The Conflict Graph Design supplies our randomization scheme and estimator. It is theory and simulation only, has no covariate adjustment, and takes $H$ as given. We add a learned $H$, covariate-augmented estimation, and robustness checks for when the graph available at assignment time is stale.

The clinical papers constrain the design. Mind the Performance Gap decomposed a C. difficile model's prospective AUROC drop, from 0.778 to 0.767, into two parts. Infrastructure shift, meaning differences between the live data pipeline and the research warehouse, cost 0.016. Temporal shift added 0.005. The live pipeline also failed on 10 of 356 days. That study treated patients as independent. We extend its decomposition with an interference term, and we inherit its warning that live graph data will be incomplete. MIMIC-Extract shows how treatments recorded just before an event leak the label into the features. It enforces gap times to prevent this, which our replay pipeline adopts. It models ICU patients as independent rows, which we drop.

The biomarker studies, the seven-marker melanoma signature and the lactylation signature, illustrate the broader point that retrospective association cannot show that acting on a model helps. We do not build on them directly.

Research Program

The program has three components.

Interference graph. Nodes are patient-days. Each edge carries features: shared room, shared bathroom, overlap in nurse or physician assignment, transfer adjacency, and temporal overlap. A message-passing model in the WCGCN style turns these into edge weights $w_{ij}$, the estimated effect of exposure at $i$ on outcome risk at $j$. It is trained on historical infection onsets against the outcome model's residual risk. The conflict graph $H$ links patients whose outcomes are jointly perturbed above a threshold.

Rollout design. Alerts are delivered or suppressed according to the conflict graph design on $H$. Estimation uses its Horvitz–Thompson-style estimator plus an augmented version that adds baseline risk covariates. The design targets three estimands: the direct effect of alerting a patient, the spillover effect of alerting their neighbors, and the global effect of full deployment versus none.

Gap accounting. We extend Ötles et al.'s decomposition with an interference term:

$$\Delta_{\text{total}} = \Delta_{\text{infra}} + \Delta_{\text{temporal}} + \Delta_{\text{interf}}.$$

The term $\Delta_{\text{interf}}$ is the difference in prospective AUROC between encounters whose high-interference neighbors were mostly alerted and those whose neighbors were mostly suppressed. The design randomizes this neighbor exposure, so it can be estimated.

Baselines

The naive reference point is the standard prospective protocol: alerts randomized individually 1:1, outcomes compared by difference in means, AUROC computed on all live encounters. This is what a careful study in the style of Mind the Performance Gap would do if it added randomization. Its reported prospective AUROC of 0.767, against 0.778 retrospectively, is our anchor for the accuracy side. We run it on the same replayed cohorts, outcome simulators, and alert budgets as every other arm.

The strongest existing competitor for estimating effects under interference is ward-level cluster randomization, in which all patients on a ward are alerted or none are. It is standard in infection-control trials and handles interference exactly when interference stays inside wards. We found no reported result on Lacuna for cluster-randomized alert trials. We will give it the same patient count and alert budget, and choose cluster boundaries with the best available administrative unit.

The second strong competitor is the conflict graph design using the administrative graph as $H$, with no learning. It isolates whether the learned graph matters. A further variant uses the observed graph with CauGramer's attention-based estimator applied after the fact. It tests whether a better estimator can substitute for a better design. Its only reported numbers, the 28–80% error reductions, come from social-network benchmarks, so we re-run it on our clinical replays.

We also ablate the Map2Schedule-style edge features: learned weights versus distance-only and room-only weights.

The proposal beats the baselines if it meets two conditions on semi-synthetic replays. First, its 95% intervals for direct and spillover effects keep at least nominal coverage where individual randomization loses coverage. Second, its interval width is materially smaller than cluster randomization's at equal $n$.

Why not just…?

Why not just cluster-randomize by ward? This is the most serious alternative, and it may win. If transmission and staff spillover stay mostly within wards, cluster randomization is unbiased, simple, and acceptable to institutional review boards. Its weaknesses are few clusters, which means wide intervals, and leakage through floating staff and transfers. We will measure how much interference actually crosses ward boundaries in the replayed contact graphs. If the cross-ward fraction is small and cluster intervals are competitive, we will recommend cluster designs and report the learned graph only as a diagnostic.

Why not just add colonization pressure to the risk model? Colonization pressure is a count of infected neighbors used as an input feature. Adding it may well improve prediction, and we will include it in every outcome model. It does not fix validation. It changes what the model predicts, not the fact that alerts alter neighbors' outcomes and contaminate both the labels and the control group. The interference term $\Delta_{\text{interf}}$ is estimated with and without this feature to show the bias persists.

Why not just learn interference from observational data with CauGramer and skip randomization? Observational hospital data mix up treatment with the reasons for treatment: clinicians isolate sicker patients, which is confounding by indication. MIMIC-Extract documents how treatment timing leaks into labels. CauGramer's negative-control variant needs domain-chosen control pairs, which are hard to justify here. The replay arm comparing it to randomized designs, under simulated confounding, shows how large this gap is.

Why not just use bed distance as the graph? Map2Schedule's urban result, where distance-based scheduling reached about 50% of capacity, shows how distance fails when structure decouples it from interaction. In hospitals, shared nursing assignments play the role buildings play in the city. Our edge-feature ablation tests whether this holds clinically. If distance-only weights do as well as learned ones, we adopt distance.

Proposed Experiments

The first stage is a semi-synthetic replay on MIMIC-III, processed through the MIMIC-Extract pipeline with its gap-time rules. We build contact graphs from the records of admissions, transfers, and care units. On these real topologies we simulate transmissible outcomes with known direct and spillover alert effects, swept across interference strength and across the fraction of interference that crosses wards. This mirrors CauGramer's semi-synthetic protocol on clinical rather than social graphs. The metrics are bias, interval coverage, and interval width for each estimand, plus the decomposed AUROC gap.

The second stage adds deployment realism. We corrupt the graph and features available at assignment time in the way Ötles et al. observed: missing static fields, delayed transfers, and roughly 3% of days without any feed. This is the clinical version of the imperfect channel information that UWMMSE flags as open. We then test whether design validity survives a stale $H$.

The third stage is a written protocol for a silent-then-live C. difficile alert pilot. The silent phase runs the model and records scores without showing alerts, which establishes the infrastructure term before any alerts reach clinicians.

Two decisive checks govern the program. The first asks whether, at plausible spillover magnitudes, individually randomized validation produces bias in the benefit estimate or in AUROC larger than the 0.016 infrastructure shift Ötles et al. measured. If it does not, the premise is weak for this setting and we say so. The second asks whether the learned $H$ beats both the administrative graph and ward clusters on interval width at nominal coverage. If it does not, the simpler design is the recommendation.