Who Counts as a Similar Patient? Denoised Similarity for Population-Level Multimodal Comorbidity Prediction

Many clinical prediction models now borrow information from similar past patients. They connect each new patient to a set of "neighbors" and let those neighbors' records shape the prediction. The idea works well when the notion of "similar" is clinical. It works badly when similarity is computed from chest X-ray features that encode where and how the image was taken: a portable bedside film versus a standing one, a visible breathing tube, ECG leads, text markers, or body habitus. Neighbor-sharing methods can then amplify a known weakness of imaging models, their habit of latching onto non-diagnostic pixels. A shortcut that is only a weak cue inside one patient's image becomes a structural bias once it decides whose records every patient borrows from.

We propose to separate two questions that current systems merge. The first is what does this patient's image show, and for that the model may use the whole image, including context outside the lungs. The second is which past patients is this one similar to, and for that the model uses only an anatomically restricted, artifact-suppressed image embedding together with the electronic health record (EHR). On the MIMIC-IV ICU stays paired with MIMIC-CXR radiographs, across 25 co-occurring diagnostic phenotypes, we test whether denoising the similarity space alone improves multi-label comorbidity prediction. We also test whether it lets the roughly three quarters of stays that have no X-ray borrow imaging evidence from clinically similar imaged patients, and whether it reduces shortcut leakage under acquisition shift.

Core Claim

Population-level models inherit their failure modes from the space in which they measure similarity. When that space is built from raw images, neighbor sharing spreads acquisition and habitus shortcuts across the cohort. Restricting the similarity embedding to the cardiothoracic region and training it to be uninformative about view position and support devices should give two results. Neighbor-based prediction should improve, especially for patients without imaging and for conditions whose evidence is spread across both data types. Each patient's own prediction pathway can still keep full-image context.

This decoupling is also a principled answer to a tension in the imaging literature. SDFN pairs a full-image classifier with one trained on U-Net lung crops and finds that the fused model (mean AUC 0.815 on ChestX-ray14) beats lungs alone (0.807). That result argues that tissue outside the lungs carries real signal. Gare et al. find the opposite in lung ultrasound: unmasked models predict COVID-19 severity from subcutaneous fat thickness (AUC 0.6553), and masking to the pleura raises AUC to 0.7742. Our position is that both are right about different jobs. Extra-organ anatomy can help describe a patient. It is a hazard when it is used to decide who that patient resembles, because similarity errors compound across neighbors.

The closest existing work differs in specific ways. PM²F²N builds a patient graph from Jaccard similarity of extracted note entities and passes messages over it for mortality and length of stay, but it has no imaging and predicts single targets. Parisot et al. use phenotypic metadata as edge weights over neuroimaging nodes for binary diagnosis. GRASP clusters patients into learned cohorts to avoid the fixed-K neighbor problem, again on EHR alone. MSRL is the nearest in spirit: it learns a cross-patient graph and keeps a buffer of real genomic features so that image-only cancer patients can borrow the missing modality. It works in pathology and genomics, with no concern for what the similarity space encodes. On the imaging side, MedPromptX crops regions of interest with an open-vocabulary detector and retrieves four nearby examples as prompts, but its retrieval embedding is not denoised, and it predicts conditions independently. None of these asks whether the similarity function itself carries shortcuts.

Research Program

The system has three parts.

The first part is the per-patient encoders. An EHR time-series encoder and a full-image chest X-ray encoder are combined by a standard fusion head. We reuse the encoder choices from the CareBench protocol so that differences come from the population component and not from backbones.

The second part is a separate similarity encoder. Its image branch sees only a dilated cardiothoracic mask, produced by a U-Net lung segmenter trained on JSRT as in SDFN and extended to cover the cardiac silhouette and the costophrenic angles. The extension addresses the documented boundary-clipping risk for effusion and cardiomegaly. The masked features are pooled with mask-weighted global average pooling, the method Zhang et al. used to suppress chest wall and air in CT. The image branch is then trained with an adversarial head, connected through gradient reversal, that tries to predict view position (AP versus PA) and the presence of support devices from the embedding. Both are available as MIMIC-CXR metadata or report-derived labels. The adversary penalizes any similarity embedding from which acquisition setting can be read off. The EHR branch of the similarity encoder is always present, so stays without an image still receive a comparable embedding. Acuity is meant to enter similarity through vitals and labs, not through the pixels of an endotracheal tube.

The third part is neighbor sharing. Each patient is connected to training-set patients by a sparse nearest-neighbor graph in the similarity space. Message passing is inductive: new or test patients attach to the training graph without retraining, and test labels never enter the graph. Messages carry two things. One is neighbors' full multi-label phenotype vectors, so that co-occurrence patterns such as heart failure together with COPD and pneumonia propagate jointly instead of one label at a time. The other, for patients who lack an X-ray, is the neighbors' masked lung-region image embeddings, which serve as borrowed imaging evidence. Following DrFuse, which showed that how useful each modality is varies sharply by condition, the weight given to borrowed imaging evidence is learned separately for each phenotype.

The analysis side matters as much as the model. For any similarity space, we measure shortcut homophily: how often an edge links two patients with the same view position or device status, compared with how often it links patients who share phenotype labels. This turns the core claim into a quantity that can be checked directly, before any downstream accuracy is reported.

Baselines

The naive reference points are a per-phenotype prevalence predictor and a unimodal EHR Transformer. CareBench found the EHR Transformer highly competitive in its Base Cohort of 26,947 stays with about 75% of chest X-rays missing, where it beat several naive multimodal models. Any population method that fails to beat it on the Base Cohort has not justified its imaging branch.

The strongest isolated fusion methods are MedFuse and DrFuse. MedFuse fuses an LSTM EHR encoder with a ResNet-34 image encoder through a recurrent fusion module that tolerates missing images. It reports 25-phenotype AUROC 0.768 and AUPRC 0.429 on its partially paired test set. DrFuse splits representations into shared and modality-specific parts and reports an 8% relative PRAUC gain over MedFuse and MMTM. No absolute DrFuse number was reported in the material we have. CareBench reports overall phenotyping AUPRC rising from 0.472 to 0.495 in its best multimodal cases. These figures come from different cohorts and splits and are not directly comparable. We will therefore rerun MedFuse, DrFuse, and the EHR Transformer ourselves on the CareBench Matched (7,149 stays) and Base cohorts. Every method gets the same encoders where applicable, the same splits, the same number of hyperparameter trials, and five seeds. Beating the field means a macro-AUPRC gain over the best rerun isolated model on the Base Cohort whose paired-bootstrap confidence interval excludes zero.

The key ablation is the identical population model with similarity computed from raw full-image embeddings plus EHR, the natural extension of the PM²F²N and Parisot designs to radiographs. This isolates the proposal's ingredient. A second ablation removes the adversarial head but keeps the mask, and a third keeps the adversary but drops the mask. These separate spatial restriction from acquisition invariance.

Two further baselines cover the remaining design choices. An SDFN-style dual-pathway isolated model tests whether masking helps without any neighbor sharing. A plain nearest-neighbor retrieval model, which averages neighbors' labels and imputes neighbors' mean image embedding with no learned message passing, is run in both raw and denoised similarity spaces.

Why not just…?

Why not just mask the image inside a standard isolated fusion model? This is a real alternative, and the dual-pathway baseline tests it directly. Our prediction, based on SDFN's finding that lung-only crops underperform full-image fusion, is that masking the prediction pathway gives small or mixed gains and hurts conditions with evidence outside the lungs. Masking only the similarity pathway should help more. If the isolated masked model matches the population model, the population component is unnecessary and we will report that.

Why not just retrieve K nearest neighbors instead of training a graph network? This may well capture most of the benefit. GRASP's argument about fixed-K neighbors and MSRL's gains over plain retrieval suggest that learned aggregation helps, but neither tested retrieval in a denoised space. For this reason the core claim is stated about the similarity space, not the aggregator. We run retrieval and message passing in both raw and denoised spaces. If denoised retrieval matches denoised message passing, the contribution becomes the similarity encoder alone, which is simpler and still new.

Why not just learn the graph structure end to end and let it discover clinical similarity? View position and device presence are predictive of ICU outcomes within the training distribution, so a graph learned to maximize training accuracy has every incentive to keep them. Nothing in MSRL's objective discourages this. We include a learned-structure variant without the adversary and measure its shortcut homophily. We expect it to sit close to the raw-image graph.

Why not just drop imaging and use the EHR Transformer? CareBench shows this is hard to beat on its Base Cohort. It also shows that for conditions whose evidence is spread across both data types, such as congestive heart failure and COPD, EHR-only models hit a ceiling that only imaging evidence broke. Our test is whether borrowed neighbor imaging lifts those specific phenotypes for stays that have no X-ray. If it does not, the EHR-only conclusion stands for that subgroup.

Proposed Experiments and Decisive Checks

All experiments use the CareBench Matched and Base cohorts over MIMIC-IV and MIMIC-CXR, with the 25 phenotypes and in-hospital mortality. We report macro AUROC and AUPRC, broken out by CareBench's split between conditions whose evidence is spread across both data types and conditions whose evidence is localized.

The first decisive check runs before any accuracy is reported. In the raw-image similarity space, shortcut homophily should be clearly elevated: edges should match on view position or device status well above chance once label agreement is controlled for. The denoised space should bring this close to chance while keeping or raising phenotype label agreement among neighbors. If the raw space shows no excess homophily, the motivating premise fails and the project reduces to a missing-modality study.

The second check is an acquisition-shift split. We train on a cohort skewed toward portable AP films and test on stays imaged PA, then repeat the split stratified by support-device status. The proposal predicts that the raw-similarity population model degrades more than isolated fusion under this shift, while the denoised-similarity model degrades less than both.

The third check concerns missing imaging. We restrict evaluation to Base Cohort stays without an X-ray. On these stays, borrowed denoised imaging evidence should raise AUPRC on heart failure and COPD above the EHR Transformer. Borrowed raw evidence should give a smaller gain or none.

The fourth check concerns fairness. CareBench reports that multimodal fusion can widen performance gaps across racial groups. We report sensitivity gaps by race for every method, and we test whether removing habitus and acquisition cues from the similarity space narrows these gaps or simply leaves them unchanged.

Finally, we run the full grid of ablations (mask only, adversary only, both, raw), crossed with retrieval versus message passing. These results determine which of the claims survive.