Diseases rarely come alone. Heart failure brings fluid in the lungs, COPD shows up alongside other chronic conditions, and how likely two diagnoses are to appear together depends on the patient: their age, their kidney function, the trend in their cardiac biomarkers. Computer vision has long used this kind of "which labels go together" knowledge, called co-occurrence modeling, to clean up ambiguous predictions. In medicine, however, those co-occurrence tables have always been computed once over the whole training population and applied identically to every patient. Separately, work on combining images with electronic health records (EHR) keeps running into two problems. Most ICU stays have no chest X-ray, and when both are present, models learn to lean on the rich EHR stream and ignore the image.
We propose to split the problem along the line where the data actually lives. First, we learn a patient-specific model of how the 25 ICU phenotypes co-occur, conditioned only on EHR trajectories. Because this model needs no images, it can be trained on every ICU stay. Second, we train an image-only model on the smaller set of stays that have X-rays, and plug its output in purely as per-label evidence that shifts the patient-specific prior. This design makes it hard to ignore the image: the image part is trained on images alone, so it cannot shortcut through EHR features. When there is no image, the model falls back to the EHR prior with no imputation. The pairwise co-occurrence terms then pass image evidence for one condition on to correlated conditions, with the strength of that coupling depending on the patient's physiology.
Core Claim
A structured joint model over phenotype labels can be written as an energy (a score over whole label sets). Its label-to-label coupling terms are functions of the EHR, and it can be trained on the full cohort, including the roughly three quarters of stays without images. We claim that such a model, combined with separately trained image evidence, will match a strong EHR-only model when images are missing and beat the best existing fusion methods when they are present. The expected gains are largest on conditions whose evidence is split across both modalities. Concretely, for phenotype labels $y \in \{0,1\}^K$ with $K=25$, EHR history $e$, optional image $x$, and hours $\Delta t$ since that image was taken, we model
$$ \log p(y \mid e, x) = \sum_{k} \big[a_k(e) + g_k(e,\Delta t)\, s_k(x)\big]\, y_k + \sum_{j<k} W_{jk}(e)\, y_j y_k - \log Z(e,x). $$
The terms are as follows. $a_k(e)$ is an EHR-derived baseline log-odds for phenotype $k$. $W(e) = W_0 + U\,\mathrm{diag}(h(e))\,U^\top$ is a low-rank, patient-conditioned coupling matrix, where $W_0$ is a population co-occurrence matrix, $U$ holds a few shared "comorbidity directions," and $h(e)$ is a small EHR encoder that scales each direction. $s_k(x)$ is the image-only classifier's log-odds with the image cohort's base rate subtracted, so that it acts as a likelihood ratio rather than a second prior. $g_k(e,\Delta t) \ge 0$ is a learned gate on how much to trust the image for phenotype $k$ given its age. $Z$ normalizes. A missing image simply sets the gated evidence term to zero.
Relation to Prior Work
Two lines of medical co-occurrence work motivate the structural choice but stay image-only and population-static. AnaXNet runs a graph convolutional network over 18 anatomical regions, with edges set by Jaccard co-occurrence of disease across regions, and reaches 0.93 average AUC on Chest ImaGenome against 0.89 for a global DenseNet. RadKG recasts CheXpert labeling as link prediction between image and finding nodes (83.5% AUC versus 82.6% for DenseNet121). Neither uses EHR, and in both the graph is the same for every patient. The natural-image method in Improving Multi-label Recognition using Class Co-Occurrence Probabilities supplies the closest technique, refining CLIP logits with a graph built from a global conditional-probability matrix, but its matrix is again fixed and label-only. DCL does build a per-case graph, but it retrieves that graph through image similarity to past reports and targets report generation. Our coupling is conditioned on structured physiology instead, and targets phenotyping. The classical co-occurrence literature also constrains us: Barnea and Ben-Shahar show that context fixes category confusions but not localization errors. We therefore apply co-occurrence only at the level of patient labels, where it is known to help, and not to regions.
On the clinical side, CareBench supplies the evidence for the gap and the evaluation setting. Its findings are that fusion models overuse EHR, that gains concentrate in CHF and COPD with negligible gains for sepsis, and that naive fusion often loses to an EHR-only Transformer in the Base Cohort, where 75% of stays lack an X-ray. Our design answers each finding. The image term is trained without access to EHR. The per-phenotype gate $g_k$ is free to shrink to near zero for localized conditions such as sepsis, which gives a direct readout of CareBench's distributed-versus-localized distinction. The EHR part is trained on all stays. DDL-CXR handles stale images by generating an updated latent X-ray with a diffusion model conditioned on the old image and the intervening EHR. We test the much cheaper alternative of discounting stale image evidence at the label level through $\Delta t$ in the gate. We expect this to be weaker than generation when the old image is still informative, and we measure the gap. MedPromptX and PECon represent the other fusion styles, serialized-text prompting and supervised contrastive alignment. Neither models joint label structure.
Research Program
We build three components and train them in stages. The first is an EHR encoder: a Transformer over hourly vitals and labs, following CareBench's preprocessing, with heads for $a(e)$ and $h(e)$. It is trained jointly with $W_0$ and $U$ on all Base Cohort stays using pseudo-likelihood, which is exact and cheap. Pseudo-likelihood means maximizing each label's probability given the other labels, and each such conditional is a simple logistic term. The second is an image-only phenotype classifier on the matched stays; its base-rate-corrected logits become $s(x)$. The third stage is a short joint fine-tune of the gates $g$, and optionally of $s$, on matched stays, with the EHR prior frozen or given a lower learning rate. Inference uses mean-field updates, which are easily tractable for 25 labels, and we check them against Gibbs sampling on a subset.
We then analyze three things. The first is whether the learned couplings $W(e)$ recover clinically sensible, patient-dependent structure, for example CHF–renal-failure coupling rising with creatinine trend. The second is whether the gates reproduce CareBench's split between distributed and localized conditions. The third is whether, as the image ages, the gates move trust from image to EHR.
Baselines
The naive reference points are a population-prevalence predictor and the EHR-only Transformer from CareBench, run on identical splits and features. The EHR-only model is the bar that, per CareBench, many fusion methods fail to clear in the Base Cohort. Our model must be statistically indistinguishable from it on stays without images, and better on stays with them.
The strongest existing competitor is the best fusion method in CareBench, together with the missingness-aware methods it highlights, MedFuse and DrFuse. For phenotyping, CareBench reports AUPRC improving from 0.472 to 0.495 between baseline and best multimodal fusion. We take 0.495 as the number to beat, and we will re-run these methods with CareBench's code on our splits rather than rely on the quoted figure. For fairness, all methods share the same image backbone initialization, the same EHR encoder architecture, and an equal hyperparameter-search budget measured in GPU-hours, with the same model selection on validation macro-AUPRC. Beating this baseline means a macro-AUPRC gain whose bootstrap confidence interval over patients excludes zero on the matched cohort, with no loss in the Base Cohort.
For asynchronicity we compare against DDL-CXR on its long-stay cohort. It reports 0.822 AUROC for mortality versus 0.815 for EHR-only. The file gives no per-phenotype number for DDL-CXR, so we re-run it for phenotyping. Our target there is parity at a fraction of the compute, not a clear win.
Two ablations isolate the key ingredient. The first, and most important, keeps everything but sets $W \equiv 0$. This leaves an independent product of an EHR model and an image model, and shows whether the co-occurrence structure does any work beyond additive logit fusion. The second replaces $W(e)$ with the static $W_0$, as in AnaXNet or the ICPR co-occurrence method, and shows whether patient conditioning matters beyond population co-occurrence. A third ablation trains the EHR prior only on matched stays, and measures how much of the gain comes from using image-less stays.
Why not just…?
Why not just use late fusion or a missingness-aware fusion network? These are the obvious strong options, and CareBench documents their weaknesses: they overuse the EHR stream, and naive variants collapse under 75% image missingness. What matters for us is data, not architecture. Late fusion can only learn from stays that have both modalities, whereas our prior learns from all of them. The "matched-only prior" ablation tests this directly. If MedFuse or DrFuse match us even after being given the same image-less stays through their missingness routing, our advantage is not the factorization, and we will report that.
Why not just add the EHR model's logits to the image model's logits? This is the real threat. It already gets the missing-image fallback and image-only training for free, and the $W \equiv 0$ ablation is exactly this model. The co-occurrence terms should help mainly in three places: rarer phenotypes that ride on common ones, the coherence of the predicted label set as a whole, and calibration of joint events. If per-label AUPRC is unchanged and joint metrics also fail to improve, the honest conclusion is that additive fusion is enough. The proposal would then reduce to a clean empirical case for it on CareBench, which is still worth publishing.
Why not just prompt a medical vision-language model with the image and the serialized EHR? MedPromptX shows that Med-Flamingo needed added visual grounding and retrieval of similar cases to reach 0.686 F1 on 968 records. The ICPR co-occurrence study reports that explicit co-occurrence refinement beat pure CLIP prompting by as much as 14.1% mAP on long-tailed data. Neither result shows that prompting handles 25 correlated labels over tens of thousands of stays with continuous time series. We include a prompted baseline on a subsample for reference, but compute cost makes it impractical at CareBench scale.
Why not just generate an up-to-date X-ray as DDL-CXR does? Generation may well be more accurate when the old image still carries anatomical information. Our label-level discount is a cheaper hypothesis, and the DDL-CXR comparison, stratified by time since imaging, will show where each wins.
Proposed Experiments
The main experiments use CareBench's Matched Cohort (7,149 stays) and Base Cohort (26,947 stays) derived from MIMIC-IV and MIMIC-CXR. We report macro- and per-phenotype AUPRC and AUROC, expected calibration error, and a joint measure: the log-likelihood of the full true label set, estimated by sampling. CareBench found that fusion can widen gaps in true positive rate across racial groups, so we also report those gaps. A secondary test uses the DDL-CXR long-stay cohort, stratified by $\Delta t$.
The decisive checks are four. First, on stays without images, our model must not fall below the EHR-only Transformer. Second, on matched stays, macro-AUPRC must exceed the re-run best CareBench fusion, and the full model must beat the $W \equiv 0$ ablation on at least the joint likelihood and on rare phenotypes. Third, the learned gates should come out large for CHF and COPD and near zero for sepsis, without being told this. Fourth, the gains over static $W_0$ should concentrate in patients whose EHR trajectories are far from the population average. If the second check fails, the co-occurrence claim fails. If the first fails, the missingness claim fails.
References
- AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-Ray
- A Relational-learning Perspective to Multi-label Chest X-ray Classification
- Improving Multi-label Recognition using Class Co-Occurrence Probabilities
- Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
- Exploring the Bounds of the Utility of Context for Object Detection
- When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
- Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation
- MedPromptX: Grounded Multimodal Prompting for Chest X-Ray Diagnosis
- PECon: Contrastive Pretraining to Enhance Feature Alignment Between CT and EHR Data for Improved Pulmonary Embolism Diagnosis