Letting the Chart Tell the Scanner Where to Look: EHR-Conditioned Local Pooling for 3D CT

Hospitals routinely pair a 3D CT scan with a patient's electronic health record (EHR), the structured history of labs, vital signs, diagnosis codes and medications. Current models combine the two streams in a weak way. Each stream is first reduced to a single summary, either a probability or one vector per scan, and only then are the summaries merged. For 3D imaging this reduction is costly. A recent benchmark of frozen 3D CT foundation models found that pooling a whole scan into one global vector "washes out" small, low-contrast findings, making them close to invisible (Big, Bright, or Invisible). Small clots in peripheral lung arteries are exactly this kind of finding. The clinician reading the scan works differently. A rising D-dimer, a low oxygen saturation, or a history of heart failure tells them which vascular beds and which heart chambers to inspect closely.

We propose to build that behavior into the model. The CT volume is kept as a variable-length set of local 3D sub-volume tokens, never resampled to a fixed depth. The patient's EHR is encoded into a small number of query vectors. These queries decide which sub-volumes are pooled into the image representation that reaches the prediction head. The claim is narrow and testable: clinical context should help most where global pooling hurts most. That means focal findings, such as subsegmental pulmonary embolism, and prognosis that depends on regional anatomy, such as later pulmonary hypertension. It should help little on large central clots that any pooling scheme already sees.

Core Claim

Existing CT-plus-EHR systems on pulmonary embolism (PE) data fuse late, align only global embeddings, or give up 3D context altogether. INSPECT trains imaging and EHR models separately and blends their probabilities. This raised PE diagnosis AUROC from 0.721 to 0.771 but gave no significant gain over the EHR-only MOTOR model for 1-month mortality, which reached 0.923 (INSPECT). RadFusion averages a 3D CNN's output with an ElasticNet's (RadFusion). PECon aligns a frozen 3D CNN's global features with an EHR projection through supervised contrastive learning. It reaches AUROC 0.943 on RadFusion, but its authors had to drop subsegmental PE cases from training because those cases hurt performance (PECon). Zhi et al. obtain high accuracy by cutting the scan into independent 2D slices and concatenating them with a 5-dimensional compressed EHR vector (Multimodal Diagnosis for PE). CT2RepLong is the closest architectural relative because it uses cross-attention to bring prior scans and prior reports into a 3D CT model. It never consumes structured EHR, and it targets report generation rather than outcome prediction (CT2Rep).

What is new here is the location of the fusion. Structured clinical data controls the spatial pooling step inside the image pathway. It does not merely add a parallel vote. The PECon and Chevli et al. results give a specific, falsifiable prediction. If the bottleneck really is global pooling of focal signal, EHR-conditioned local pooling should recover subsegmental PE sensitivity. Unconditioned global pooling at matched capacity should not.

Research Program

The image pathway keeps both 3D context and variable depth. We tile each CTPA volume at its native slice spacing into overlapping thin slabs, each a few millimetres deep. A 3D convolutional stem encodes each slab, so there is real inter-slice context of the kind 3D kernels gave at ambiguous boundaries in 3D Deeply Supervised Network. An inter-slab transformer along the depth axis then mixes the slab tokens, following CT2Rep's causal inter-slice design. The number of tokens varies with scan length. This avoids the padding and interpolation that Zhi et al. blame for poor 3D CNN results on scans ranging from 28 to 2,600 slices.

The clinical pathway encodes the pre-scan EHR with a pretrained longitudinal model. We use MOTOR, as released with INSPECT, which avoids the overfitting Zhi et al. saw when 2,912 raw variables went into an MLP and produced 52.2% accuracy. A small head maps the patient embedding to $K$ query vectors, with $K$ around 4 to 8. Each query attends over the slab tokens, and the $K$ attended summaries form the image representation. The final predictor sees two inputs: this conditioned image summary and the EHR embedding, which enters through a separate residual path.

That separate path is deliberate. The cheapest way to exploit the EHR is to bypass the image entirely, and if that path is open we want it to be explicit and measurable rather than hidden inside the attention. We will report how much each pathway contributes. We also train the image summary with an auxiliary image-only loss, so that the image branch cannot fall back on simply copying the EHR.

The backbone is fine-tuned end to end. PECon kept its visual encoder frozen, so the clinical signal never shaped the spatial filters, and lifting that restriction is part of the test. We will also train with subsegmental cases included, which PECon could not do.

Baselines

The naive reference points are the two unimodal models. The first is EHR-only MOTOR, which on INSPECT reports AUROC 0.923 for 1-month mortality and 0.824 for pulmonary hypertension (INSPECT). The second is image-only, using our own slab encoder with mean pooling. All models use the same patient-level splits, the same EHR encoder weights, and the same hyperparameter search budget.

The strongest existing fusion method on INSPECT is weighted late fusion, which reports diagnosis AUROC 0.771 (0.761 in the arXiv version). We will re-run it with our image encoder, so that any gain cannot be credited to a better backbone.

On RadFusion, the strongest reported competitor is PECon, at AUROC 0.943, F1 0.913 and accuracy 0.90. We will reimplement it on our splits, once with its frozen encoder and once fine-tuned. We will also run the Zhi et al. 2D-slice-plus-MDS model. Its reported 97.3% accuracy comes from a 600-patient subset with a 120-patient test set, so we will re-evaluate it on full RadFusion rather than compare against that figure.

The ablation that isolates the key ingredient is attention pooling with $K$ learned queries that do not depend on the EHR, followed by the same EHR residual path. This model matches ours in parameters, compute and local-pooling capacity, and differs only in whether clinical context steers the pooling. We will also run our model with EHR queries taken from a randomly shuffled patient.

Beating the field means three things. First, a statistically significant AUROC gain over the unconditioned-query ablation on subsegmental PE, tested by bootstrap over patients. Second, no loss on central PE. Third, on INSPECT, improvement over 0.771 for diagnosis and over EHR-only 0.824 for pulmonary hypertension.

Why not just…?

Why not just use late fusion of strong unimodal models? Late fusion is modular and robust to missing data. It is the right baseline for diagnosis, where it already helps. However, INSPECT shows it adds nothing for prognosis, because a scalar probability cannot carry information about where in the scan the imaging evidence lies. Our late-fusion re-run uses the same image encoder. If it matches our model on the subsegmental and pulmonary hypertension subsets, the proposal fails, and we will report that.

Why not just replace global pooling with attention pooling and leave the EHR out of it? This is the most serious threat. The Chevli et al. finding concerns pooling, not clinical data, and generic attention-based pooling over local tokens might recover focal signal by itself. That is why the unconditioned-query ablation is the central comparison rather than a footnote. If it closes most of the subsegmental gap, the honest conclusion is that local pooling is what matters and that EHR conditioning is unnecessary. We would then present the paper that way.

Why not just slice into 2D, as Zhi et al. did? Their result is real but comes from a small subset. Their 3D baseline was handicapped by forced depth resampling, which our variable-length slab tokens avoid. We will run their pipeline on full RadFusion and on INSPECT. If 2D slices match 3D slabs inside our conditioned model, we will adopt 2D slabs and keep the conditioning. The depth of each slab is an ablation axis for exactly this reason.

Why not just use EHR alone, given that it dominates prognosis? For mortality it probably does. At 0.923 there is little room for imaging to add anything, and we do not expect a gain there. Pulmonary hypertension is different: it reflects regional anatomy such as right-ventricular enlargement, which the EHR records only indirectly. That task, at 0.824, is where the claim is tested. A null result there would support INSPECT's view that imaging and EHR divide the labor between acute and long-term questions.

Proposed Experiments and Decisive Checks

All experiments use RadFusion for diagnosis and fairness, and INSPECT for diagnosis plus 1-month mortality, readmission and pulmonary hypertension.

The first decisive check is a stratified diagnosis analysis. We compare the conditioned model, the unconditioned-query model and global pooling, with AUROC reported separately for subsegmental, segmental and central PE. Subsegmental labels come from RadFusion's annotations, the same ones PECon used to exclude cases. The prediction is a gain that grows as clot size shrinks.

The second check is counterfactual. We swap in the EHR queries of a patient with discordant labs, for example a normal D-dimer, and test whether the attention mass on the scan shifts and whether image-branch-only predictions change. If neither happens, the conditioning is decorative.

The third check concerns spatial plausibility. We measure attention mass by lung region, derived from a frozen segmenter such as SegVol, and compare it with the clot laterality stated in the radiology report.

The fourth check is prognosis on INSPECT pulmonary hypertension against EHR-only 0.824. We also measure attention on the cardiac region, testing whether cardiac-history queries direct pooling toward the heart.

Finally, because conditioning image attention on the EHR could import EHR biases into the image pathway, we report RadFusion's equal-opportunity difference by race and gender. The reference point is late fusion's maximum gap of 0.059 (RadFusion). A model that gains accuracy but widens this gap is not an improvement. Both datasets come from a single institution, Stanford, so all claims are limited to within-site generalization. A cross-site test would require external data that this file does not provide.