Does the Note Matter, or Only When It Was Written? A Granger Test for Clinical Fusion

Hospital records mix dense streams, such as hourly vital signs and labs, with sparse ones, such as chest X-rays and radiology reports. A sparse item appears only when a clinician decides to order or write it, and that decision usually follows a change in the patient. So when a multimodal model gains accuracy from adding X-rays or notes, there are two possible explanations. The model may be reading what the image or note says. Or it may only be registering that an image or note exists at a given hour, which is a strong sign that someone was worried. We propose to separate these two sources of information with a nested predictive test in the style of Granger causality. Granger causality is the statistical idea that $X$ "causes" $Y$ if adding $X$'s past improves forecasts of $Y$'s future beyond $Y$'s own past. We then use the test's results to decide which sparse content, at which delay, a prediction model is allowed to fuse.

The idea borrows a lesson from financial markets. Information reaches prices with a delay, and news from previous days often moves multi-day trends more than same-day headlines do (Hu et al.). Models that only look at features from the same timestamp miss these delayed effects (MASTER). Clinical documents behave the same way: a report written at hour 10 may matter most for physiology at hour 30. If the model lets content in only at the delays where it adds information beyond the dense history and beyond the timing of acquisition, it should avoid two failures documented in the clinical fusion literature. One is that the dense stream dominates and the model ignores the sparse one. The other is that performance collapses when the sparse modality is often missing.

Core Claim

We claim that much of the apparent benefit of clinical multimodal fusion comes from the observation process (when a sparse modality was acquired) rather than from its content. We further claim that a lag-resolved test can measure the two contributions separately. Let $H_t$ be the dense EHR history up to time $t$. Let $O_t$ be the observation process, meaning the timestamps and missingness indicators of every sparse item up to $t$. Let $C_{t,\ell}$ be the content embeddings of sparse items acquired in the lag window $\ell$ before $t$, with windows such as 0–6h, 6–24h and 24–72h. For an outcome $y$ at horizon $h$, we define the timing gain and the content gain as differences in held-out log-likelihood between three nested models:

$$\Delta_O = \mathbb{E}\left[\log p(y_{t+h}\mid H_t, O_t) - \log p(y_{t+h}\mid H_t)\right],\qquad \Delta_C(\ell) = \mathbb{E}\left[\log p(y_{t+h}\mid H_t, O_t, C_{t,\ell}) - \log p(y_{t+h}\mid H_t, O_t)\right].$$

All expectations are estimated on folds not used for fitting. The key design choice is that the content gain $\Delta_C(\ell)$ is conditioned on timing. That makes it a Granger-style test of content that is robust to informative missingness. In clinical data, missingness is "not at random": tests are ordered because the patient is deteriorating. Under that condition, an ordinary Granger test credits timing information to the content.

The model we build, which we call the Granger-gated fusion model, admits content embeddings only for the (modality, lag) cells whose content gain is significant across folds. It always includes the observation process. This also yields a falsifiable prediction tied to CareBench, which found fusion gains concentrated in "modality-distributed" conditions such as heart failure and COPD, and no gain for sepsis. We predict that the content gain will be clearly positive at multi-hour lags for heart failure and COPD and near zero for sepsis. For sepsis, we predict that the timing gain will carry whatever signal the sparse modality has.

Relation to Prior Work

Several papers motivate this work by showing the gap. Affect2MM runs Granger tests between six video, audio and text streams and uses them to steer cross-modal attention. However, it assumes every stream is densely and regularly observed, so it has no notion of an observation process to condition on. CRL-MMNAR shows that missingness in MIMIC-IV is informative and encodes stay-level missingness patterns into an embedding that guides fusion. It collapses each stay to a static vector, so it cannot say at what delay content matters, and it does not separate content from timing. We reuse its cross-fitting estimator, a procedure that fits on some folds and evaluates on others to remove bias, but we apply it to lag-resolved likelihood differences. PM²F²N aligns note entities with segments of vital-sign series through co-attention. It never tests whether text predicts future physiology beyond what the physiology history already predicts. CareBench supplies the cohorts, the 75%-missing setting, and the finding we aim to explain mechanistically.

Two papers supply methods. Neural Granger Causality gives the component-wise idea of testing one input group at a time with structured sparsity. Its regular-grid assumption is the reason we treat sparse items as timestamped events binned by lag rather than as grid channels. MASTER and Gaia show that separating within-series temporal encoding from cross-entity aggregation is what lets delayed coupling be learned. Our lag windows are a small, testable version of that separation.

Two papers constrain the design. Function-Valued Causal Influence warns that a single scalar edge score can hide thresholds and sign reversals. We therefore report lag profiles and condition-specific gains rather than one number. That paper also notes that its estimator degrades in sparse regions, so we deliberately avoid curve-shape claims about sparse modalities. GPlag estimates continuous delays but only for pairs of series. We accept coarse lag windows in exchange for scaling to many variables and to text and image content.

Research Program

The dense EHR history is encoded by a time-series Transformer on the hourly grid from MIMIC-Extract or its MIMIC-IV analog. The observation process is encoded from the times since each sparse item, the counts of sparse items per lag window, and indicators for which modalities are present. Content is encoded with frozen domain encoders: a clinical text encoder for radiology reports and notes, and an image encoder for chest X-rays. Each content vector is assigned to the lag window in which it was acquired relative to the prediction time.

All three nested models share the dense and timing encoders, and they get the same tuning budget per fold. Content gains are estimated with five-fold cross-fitting at the patient level, and per-cell significance comes from a patient-level bootstrap with correction for multiple lags. The gated model then refits using only the cells that pass.

We run the same three-model decomposition on news-driven stock prediction (ACL18). There, the observation process is the intensity of news arrival and the content is the news text. The purpose is to check whether the timing-versus-content split is a general property of delayed reaction to sparse text, as the market literature on delayed absorption suggests, rather than a quirk of hospitals.

Baselines

The naive reference is a unimodal dense-EHR Transformer with no sparse modality. CareBench reports that models of this kind consistently beat complex fusion models in the 75%-missing base cohort. No reported result was found on Lacuna for its exact AUROC or AUPRC, so we will reproduce it ourselves on CareBench's cohort splits. The best CareBench phenotyping result the file reports is an AUPRC rise "from 0.472 to 0.495 in the best cases" with fusion. We take 0.495 as the reference fusion number on the matched cohort.

The strongest existing missingness-aware fusion methods are CRL-MMNAR, which reports "up to a 13.8% improvement" in readmission and ICU-admission prediction, and CareBench's DrFuse and MedFuse. No reported result was found on Lacuna for absolute AUROC values of CRL-MMNAR, DrFuse or MedFuse, so all three will be rerun on identical patient splits with the same encoders and hyperparameter search budget. For MIMIC-III mortality with notes, PM²F²N reports 90.22% AUROC, and MIMIC-Extract reports 0.85–0.90 for time-series-only models. Both serve as anchors.

The ablation that isolates our key ingredient is the same gated model but with content gains computed without conditioning on $O_t$, which is the ordinary Granger test. A second ablation is full fusion with timing features included and no gate. Beating the baselines means two things. On the 75%-missing cohort, the model must at least match the unimodal Transformer and exceed the best rerun fusion method in AUPRC, with bootstrap intervals that do not overlap. On the matched cohort, it must be no worse than 0.495 phenotyping AUPRC. In addition, the subgroup true-positive-rate gaps that CareBench flagged must not widen relative to the unimodal model.

Why not just…?

Why not just use the unimodal EHR Transformer and stop fusing? This is the most serious threat, since CareBench finds it wins under realistic missingness. Our decomposition tests it directly. If both $\Delta_O$ and $\Delta_C(\ell)$ are near zero for most conditions, then the unimodal model is the right answer and our contribution reduces to a diagnostic that says so for each condition. We will report that outcome as a valid result.

Why not just add missingness indicators and timing features to a full fusion model, as CRL-MMNAR does? This may match our model in accuracy, because the timing channel is exactly what we predict carries most of the signal. The ungated-with-timing ablation settles the question. If it ties, our advantage is limited to smaller models, robustness when content is missing, and an explicit lag profile. We will not claim more than that.

Why not just let a learned soft attention gate choose lags, as in MASTER-style delayed attention? A learned gate is trained on the same likelihood, so in principle it can find the same cells. What it lacks is a held-out significance criterion, and under dominance by the dense stream it tends to drive sparse weights toward zero or toward noise (Beyond Concatenation). We will compare the two gates on accuracy and on recovery of planted lags in the semi-synthetic test below. If the learned gate recovers lags equally well, we will adopt it and keep the test as an audit tool.

Proposed Experiments

The first check is semi-synthetic. On real MIMIC-IV dense histories, we plant an outcome that depends on report content at a known 12–24h lag. We then generate report acquisition times from a deterioration-driven ordering rule, so that timing alone is predictive. The decisive pattern is that the ordinary Granger ablation flags spurious content gains at short lags, while our conditioned test recovers only the planted cell. We also vary the planted effect size to map the test's power at 75% missingness.

The second check is the CareBench prediction. On the matched cohort (7,149 stays) and the base cohort (26,947 stays), we estimate lag profiles of $\Delta_O$ and $\Delta_C$ for heart failure, COPD and sepsis. The hypothesis is falsified if sepsis shows content gains comparable to heart failure, or if heart failure's gain disappears once timing is conditioned on.

The third check is predictive. We compare the gated model against all baselines on CareBench phenotyping and on MIMIC-III mortality with notes, under the criteria stated in the Baselines section.

The fourth check is the market replication. On ACL18, where CausalStock reports 63.42% accuracy against 62.69% for CMIN (CausalStock), we decompose news gains into arrival-timing and content components across one- to five-day lags. If content gains appear only at multi-day lags, the delayed-absorption account holds in both domains. If timing explains most of the news benefit, that is a finding finance models have not tested for.