Reading Modality Decay Without Labels: Cross-Modal Agreement for Fusion Models Under Diagnostic Delay

Early sepsis, insidious cancers, and autoimmune disease all begin with vague signals: a slightly raised heart rate, an odd phrase in a nursing note, a borderline lab value. No single data source is convincing on its own. This is why combining data types helps early diagnosis. Fusing clinical notes with vital signs raised early-sepsis AUROC to 89.31% in Improving Early Sepsis Prediction with Multi Modal Learning. Hospitals change over time, though. Coding systems migrate, lab instruments get replaced, and documentation habits drift. Different data sources decay at different rates. When one source quietly goes stale, a fusion model keeps trusting it at its old weight. The usual fix is to refit on recent labeled data. That fix is weakest exactly where diagnostic latency is longest, because the confirmed diagnoses needed for refitting arrive weeks, months, or years after the patient was seen.

We propose to estimate how informative each data source still is from unlabeled recent patients, by watching how well the sources agree with one another. If a disease drives the vitals, the codes, the notes, and the imaging, each is a noisy view of the same hidden disease state. A source that has stopped tracking that state stops co-varying with the others, and this can be measured before any new label exists. The fusion weights are then re-estimated from these label-free measurements. The approach targets the case the temporal-shift literature leaves open: discrimination is lost, refitting is blocked by delayed labels, and invariance methods have already been shown to fail.

Core Claim

Diagnosis is mostly "anti-causal": the disease $Y$ produces the observations $X$, rather than the other way around. That framing is argued in Considerations for Distribution Shift Robustness of Diagnostic Models in Healthcare. In this setting, the per-modality risk scores of a late-fusion model behave approximately like a one-factor model. Write $s_m$ for the score of modality $m$ (vitals, labs, codes, radiology text, image), $z$ for the latent disease severity, and $\lambda_m$ for how strongly modality $m$ currently tracks it:

$$s_m = \lambda_m z + \epsilon_m .$$

If the noise terms $\epsilon_m$ are uncorrelated across modalities, any three modalities $m,n,k$ identify the loading of $m$ from unlabeled covariances $C_{mn}=\mathrm{Cov}(s_m,s_n)$. This is the classical triad identity of factor analysis:

$$\lambda_m^2 \propto \frac{C_{mn}\,C_{mk}}{C_{nk}} .$$

The inverse-noise-weighted combination $w_m \propto \lambda_m/\mathrm{Var}(\epsilon_m)$ then approximates the best linear fusion. Our central claim is empirical and falsifiable. On real multi-year clinical data, the drop in a modality's label-free loading tracks the drop in its labeled discrimination closely enough that re-weighting the fusion combiner from these estimates recovers a meaningful share of lost AUROC. It should do so without any post-shift labels, and in regimes where label delay makes refitting impossible.

The obvious objection is that modalities are not conditionally independent. Labs and vitals share physiology, and codes are partly written from notes. We do not assume these residual dependencies vanish. We estimate the within-class residual covariance in a labeled reference era and assume only that this dependence structure drifts more slowly than the loadings do. That weaker assumption is directly testable, and testing it is part of the program.

Research Program

Each direction plays a different role. Diagnostic latency defines the regime: weak single-modality signal plus delayed labels. Heterogeneous fusion supplies the multiple views that make label-free estimation possible. Temporal shift is the stress that the method must survive.

The model is deliberately a late-fusion design. Each modality gets its own encoder and its own calibrated risk head, trained with ordinary pooled empirical risk minimization (ERM, i.e. standard supervised training on all available data). A small linear combiner sits on top. We choose this design for three reasons. Siloed deep branches before fusion are what MUFASA found worked best on MIMIC-III. Late fusion is what Qin et al. used successfully. And it exposes per-modality scores whose agreement can be measured. We do not touch the encoders during deployment, and we do not enforce invariance. That choice is constrained by Guo et al., who found that invariant risk minimization (IRM) and unsupervised domain adaptation never beat pooled ERM on MIMIC-IV temporal shift. Their explanation is that drifting features often carry real signal, so forcing invariance removes it.

Following Considerations for Distribution Shift Robustness, demographic covariates such as age and sex are treated as causes of prevalence and conditioned on rather than removed. The factor model is fit within demographic strata, so that a change in case mix is not misread as a modality decaying.

The program has three parts. First, we measure how each modality's discrimination and loading evolve across eras, which answers the uncharacterized question of how fusion interfaces decay over time. Second, we validate the label-free estimator by correlating estimated loading changes with labeled per-modality AUROC changes era by era. Third, we deploy the re-weighted combiner under simulated label delays and compare it with refitting on whatever labels the delay allows.

We also run a diagnostic for when the method should fail. We inject a synthetic shock in which two modalities drift together, for example corrupting codes and the notes they are derived from at the same moment. Correlated decay is the case the triad identity cannot separate, and we report it rather than hide it.

Baselines

The naive reference is a frozen fusion model trained on the earliest era and never updated. On MIMIC-IV sepsis, the analogous tabular model lost 0.090 AUROC (95% CI 0.080–0.101) from 2008–2010 to 2017–2019. That is one extra missed case per 11 sepsis patients, as reported by Guo et al.. We expect our method to recover part of a comparable drop in the multimodal setting.

The strongest existing competitor is pooled ERM over all labeled history available at deployment time, the ERM[08–16] analogue. Guo et al. found that domain generalization and adaptation methods differed from it by −0.003 to 0.050 AUROC and did not reliably beat it. The file gives no absolute AUROC for that model in the summary I have, so we will reproduce it ourselves with the same windows. We run it on identical splits and modalities, with the same per-modality encoders. Beating it means a statistically significant AUROC and AUPRC gain on the final era, using a bootstrap over patients.

The second strong competitor is label-free sample reweighting from Stable clinical risk prediction against distribution shift in electronic health records. That method reweights pre-shift patients to match post-shift code frequencies and latent visit distributions, and gains about 17.2% AUROC over post-shift-only training on MarketScan CCAE. It is the closest relative of our idea because it also adapts without post-shift labels. It differs in two ways: it reweights patients, not modalities, and it works on one structured modality. We apply it to each modality's encoder training with the same compute and tuning budget, and we also test it combined with our method.

The third reference is the lag-$k$ refit combiner: the fusion weights are refit on labels that would have matured $k$ eras ago. This is the realistic reactive-maintenance practice. An oracle that refits on current-era labels provides the ceiling.

Two ablations isolate the key ingredient. The first replaces agreement-derived weights with confidence-based gating, which downweights a modality whose scores become less extreme. The second applies the factor model while ignoring the reference-era residual dependence. Both ablations are run over the same encoders.

Why not just…?

Why not just refit the fusion combiner on the latest labels you have? This is the real threat, and at short label delays it may well win. A combiner with five weights needs few labels. Our answer is not that refitting is wrong but that it learns the weights of era $t-k$. The experiment sweeps $k$ from zero to two eras and reports the crossover delay beyond which agreement-based weights match or beat lag-$k$ refitting. If no such crossover exists on our data, the claim fails. For conditions like ankylosing spondylitis, whose cohort in Zhu et al. spans 2012–2021, or the slow cancer timelines in EMDOT, $k$ is plausibly large.

Why not just recalibrate periodically? The systematic review in Systematic Review of Approaches to Preserve Machine Learning Performance found that recalibration restores calibration but not lost discrimination. Missed early diagnoses are a discrimination failure. Recalibration also needs labels. We include recalibration on top of every arm so that our measured gains are gains in ranking, not in scaling.

Why not just use one robust modality, or a simple model? EMDOT found that logistic regression aged better than gradient boosting or neural networks on SEER lung cancer. The per-modality AUROC trajectories we measure will show whether one stable modality, likely vitals, dominates every era. If it does, fusion adds nothing late in deployment, and we will say so. The best-single-modality model is reported as an explicit baseline.

Why not let learned attention gating adapt per patient? Attention weights are learned on training-era data. Nothing in their objective tells them that a modality's link to the disease has weakened after deployment. The confidence-gating ablation tests the closest label-free version of this idea.

Proposed Experiments

The primary testbed is MIMIC-IV, partitioned into the 2008–2010, 2011–2013, 2014–2016, and 2017–2019 windows of Guo et al.. We use five modalities: vital time series, labs, diagnosis and medication codes, chest radiographs from MIMIC-CXR, and their radiology reports. The tasks are sepsis and their other three outcomes. The ICD-9 to ICD-10 transition falls inside this span and provides a natural code shock. Evaluation follows EMDOT-style rolling deployment rather than random splits, because random splits hide temporal decay.

The second testbed is MarketScan CCAE around October 2015. There, diagnoses, procedures, and medications serve as separate views, which allows a direct head-to-head comparison with Stable clinical risk prediction. Cross-site checks on the PhysioNet 2019 data test whether loadings explain institutional failure. The AEC-Net ensemble in Representation Learning for Early Sepsis Prediction fell from 0.41 validation utility to 0.284 test utility. That dataset supports only numeric views, so the factor model will be fit on vital and lab subgroups there.

Metrics are AUROC, AUPRC (which matters given low prevalence), calibration error after recalibration, and the PhysioNet utility score where it applies.

Two checks are decisive. The first is the era-by-era rank correlation between estimated loading change and labeled per-modality AUROC change. If it is weak, the premise is wrong. The second is the final-era AUROC of agreement-weighted fusion against pooled ERM and against lag-$k$ refitting across the delay sweep. A positive result would show that agreement-weighted fusion beats pooled ERM and matches refitting at delays of one era or more. It would also show the gap concentrated in the modalities hit by known shocks, such as codes after the ICD-10 transition. A negative result on the correlated-shock stress test is expected and will bound where the method can be trusted.