Using Clinical Notes and Images to Untangle Opioid Harm from Patient Severity

When a hospital database shows that patients given opioids do worse, two stories fit the data. The drugs may cause the harm, or doctors may simply give more opioids to patients who are sicker and in more pain. Electronic health record studies of opioid adverse effects mostly cannot tell these stories apart. One sepsis cohort reported an adjusted 28-day mortality hazard ratio of 6.239 for opioid exposure, yet its authors noted that the repository had no pain scores at all (Prescription opioids are associated with higher mortality in patients diagnosed with sepsis). An ICU delirium nomogram treats fentanyl as a binary admission covariate and reports an odds ratio of 1.94 (A nomogram for predicting delirium in the ICU among older patients with COPD). Much of what drives a clinician to escalate opioids is never captured in structured fields. It lives in free text such as radiology reports and prior discharge summaries, in chest X-rays, and in the decision of which tests to order.

This proposal applies multimodal data fusion to a different job. Fusion here means models that combine tables, time series, text, and images. The aim is not better prediction of opioid complications. The aim is better adjustment for confounding, meaning the patient factors that drive both opioid dosing and the bad outcome. We will build a fused representation of everything known about a patient before opioid exposure. We then use it inside a cross-fitted doubly robust estimator, which combines a model of who gets treated with a model of the outcome and stays approximately unbiased if either model is right. The question is whether each added modality moves the estimated effect of early fentanyl dose on ICU delirium and naloxone rescue. We test whether those shifts are real reductions in bias, using semi-synthetic data with a known answer and negative-control checks.

Core Claim

For opioid adverse effects, the value of heterogeneous fusion is in reducing confounding by indication, not in raising AUROC. Prediction metrics cannot show this value. Two models can reach the same AUROC while carrying very different information about why a patient was treated.

We claim three things. A pre-exposure representation fused from structured time series, radiology report text, and chest X-ray embeddings, trained jointly to predict treatment and outcome, will give smaller bias than a structured-covariate propensity model on semi-synthetic benchmarks where some confounding sits only in text or images. It will not add bias when confounding is purely structured. On real MIMIC-IV data, it will move the fentanyl–delirium estimate in a direction that also pulls negative-control associations toward zero.

The closest existing work splits across two camps. Fusion papers such as MedFuse, CareBench, and CRL-MMNAR optimize outcome prediction. Opioid-harm papers estimate associations with hand-picked tabular covariates. Nobody in the file uses fusion as the confounder model for a drug-harm estimate, or evaluates fusion by bias rather than discrimination.

Research Program

Cohort and estimand. The cohort is adult MIMIC-IV ICU stays that have at least 24 hours of data and no delirium on the first CAM-ICU assessment. The exposure is cumulative fentanyl dose in the first 24 ICU hours, expressed in morphine milligram equivalents and split at a clinically chosen threshold. A continuous-dose version is a secondary analysis.

The primary outcome is a positive CAM-ICU between hours 24 and 96. The secondary outcome is naloxone administration within the same window, used as an imperfect proxy for opioid-induced respiratory depression. The estimand is the average treatment effect on the risk-difference scale.

Time windows are strict. Every covariate, note, image, and test-ordering event must come before or within the first six hours. This prevents adjustment for consequences of opioid exposure, which would bias the estimate. Discharge summaries from the current stay are excluded for the same reason. Only summaries from prior admissions, plus radiology reports and images timestamped before exposure, enter the model.

Modalities. The model receives four kinds of input. The first is static and structured data: demographics, comorbidity indices, SOFA, GCS, and the LASSO-selected variables of Yu et al. The second is the irregular early vital-sign and lab series, including charted pain scores where they exist. The third is pre-exposure radiology report text, encoded by a frozen clinical language model. The fourth is MIMIC-CXR image embeddings.

Missingness is common and informative. CareBench reports roughly 75% of stays without images, and CRL-MMNAR reports 24.5% of patients without discharge summaries. Following CRL-MMNAR, we encode each missingness pattern as its own embedding. Here that is defensible as a confounder, because a pre-exposure order for an emergency chest film is a severity signal the clinician acted on.

Fusion architecture. Image vectors overwhelm low-dimensional tables during training when the two are simply concatenated (TabMixer). To avoid this, we project every modality to a shared low dimension. The time-series encoder is an LSTM in the MedFuse style, and the other modalities enter as gated additions, so a missing modality contributes nothing. The shared representation feeds two heads, one predicting treatment and one predicting outcome. This follows the logic of joint propensity–outcome representation learning: features that predict treatment but not outcome still matter for adjustment.

Estimation. The two heads supply the propensity and outcome estimates for an augmented inverse-propensity-weighted estimator with five-fold cross-fitting. Cross-fitting means the nuisance models are always fit on folds other than the ones they score, which removes overfitting bias. This reuses the cross-fitting machinery of CRL-MMNAR. The difference is that CRL-MMNAR applies it to correct its predictive representation for missingness, whereas we use it to estimate a treatment effect.

We report overlap diagnostics and trim propensities outside [0.05, 0.95]. Trimming matters because richer covariates can sharpen propensities and destroy overlap. That failure mode is real, and we will measure it rather than hide it.

Secondary question. Heterogeneity by age, insurance, and race is examined as a secondary analysis. Measurement Bias in ML-Enhanced Opioid Risk Scoring shows that opioid risk features proxy for Medicare status and age, and CareBench shows that fusion can widen true-positive-rate gaps. We therefore check whether the subgroup effect estimates from fused adjustment stay stable or become noisier in under-represented groups.

Baselines

Unadjusted reference. The naive point of comparison is the unadjusted risk difference in delirium between high- and low-fentanyl stays. Every adjusted method should move away from it. Its distance from the adjusted estimates shows how much confounding exists at all.

Structured clinical adjustment. The strongest existing approach is a propensity and outcome model built on the structured covariate set of Yu et al.. That set covers oxygen flow, cerebrovascular disease, comorbidity index, GCS, SOFA, heart rate, temperature, BUN, and urine output. We add charted pain scores and summary statistics of the early vital-sign series.

We fit this baseline two ways: with logistic regression, and with gradient-boosted trees. Gradient-boosted trees reached an AUC of 0.851 for postoperative delirium versus 0.763 for a clinician-guided logistic model in Bishara et al.. Both versions plug into the same cross-fitted doubly robust estimator, with the same folds, trimming rule, and hyperparameter search budget as the fused model.

The only reported number on this cohort is the nomogram's AUC of 0.86 and the fentanyl odds ratio of 1.94 from Yu et al. No reported result found on Lacuna for causal-effect bias of any adjustment method on opioid outcomes. Beating this baseline means lower absolute bias and better confidence-interval coverage on the semi-synthetic benchmarks, and no worse performance when confounding is purely structured.

Structured time-series encoder. This baseline isolates whether gains come from the extra modalities or simply from the neural encoder. It is the same LSTM two-head model on structured data and missingness patterns only, with no text and no images.

Concatenation ablation. The same inputs are concatenated directly without projection or gating. This tests whether the asymmetry corrections matter for adjustment as they do for prediction.

Prediction-only representation. The fused encoder is trained only on the outcome, and propensities are fit afterwards. This tests whether the treatment head, the key ingredient, is necessary.

Why not just…?

Why not just add the structured pain scores and severity scores MIMIC-IV already charts? This is the most serious threat. If charted pain, SOFA, and early vitals capture what drives opioid dosing, text and images add variance and nothing else. Our second baseline is built to test exactly this. On real data, a negative result would look like the fused and structured estimates agreeing within their confidence intervals, with negative controls no closer to zero. That would be a useful finding in its own right: structured MIMIC-IV adjustment is adequate for this question. The proposal is designed so that this outcome can be reported cleanly.

Why not just train the best possible delirium predictor, as TRAPOD and MedFuse do? A high AUROC says nothing about confounding. TRAPOD reaches a mean AUROC of 0.774 while using remifentanil infusion as an input. That tells us opioids are predictive, not whether reducing them would help. We will report predictive AUROC for every nuisance model to show that rankings by AUROC and rankings by causal bias can disagree.

Why not just extract a few keywords or language-model-derived labels from the notes and add them as tabular covariates? This could work about as well, and it would be more interpretable. We include it as a further baseline. We take terms such as "agitation," "severe pain," "aspiration," and "edema" from pre-exposure reports and add them to the structured model. If it matches the learned fusion on bias, we will recommend it instead. The learned representation is justified only if confounding signal in images, or in unanticipated text, goes beyond fixed keywords.

Why not just use the unimodal EHR transformer that CareBench found hard to beat? CareBench found that EHR-only models often beat fusion under 75% image missingness. Fusion gains appeared mainly for conditions with signs spread across organs, with phenotyping AUPRC rising from 0.472 to 0.495. Opioid indication is plausibly spread across modalities in the same way, combining pain, respiratory status, and imaging findings. Still, the CareBench result means we expect small image contributions. We will report the effect of each modality separately rather than assume fusion helps.

Proposed Experiments

Semi-synthetic benchmark. This is the decisive test. We keep the real MIMIC-IV covariates, text, and images, and simulate treatment and outcome with a known effect. We vary where the confounding lives across three settings: in structured variables only; in a latent severity score derived from radiology report findings only; and in image findings only, using labels derived from CXR reports.

To avoid circularity, the confounding signal is generated from report-derived labels that the encoder never sees during training. We sweep confounding strength and true effect size. The metrics are absolute bias, root-mean-squared error, and 95% interval coverage. Success requires two things: bias reduction in the text and image settings, and no inflation of bias in the structured-only setting.

Real-data estimation. On real data, we report the fentanyl effect on delirium and naloxone rescue under each adjustment set. We include the E-value, which measures how strong an unmeasured confounder would need to be to explain away the estimate. We also report the change in that E-value as modalities are added.

Falsification checks. We pre-specify negative-control outcomes that fentanyl should not plausibly cause within 72 hours, chosen with clinical collaborators. Examples are diagnosis codes for chronic conditions first recorded during the stay. A better confounder model should pull these associations toward zero. A model that moves the primary estimate while leaving negative controls biased is flagged as untrustworthy.

External replication. For the structured arm, we replicate on the multi-center eICU database. eICU has no images or reports, so this tests whether the structured-baseline estimate transports across 208 hospitals. It is a stability check, not a test of fusion.

The project's success criterion is explicit. It succeeds if it shows when multimodal adjustment changes an opioid-harm conclusion and when it does not. A clear negative answer to the first "why not just" question counts as a result.