Hospital prediction models increasingly combine two kinds of patient data: structured tables of vital signs, lab values and billing codes, and free-text notes written by nurses, radiologists and physicians. The notes matter because they record findings the tables never capture, such as lung sounds, mental status, or a radiologist's impression. Current fusion models, however, appear to read notes as bags of alarming words. The authors of one such system report that it cannot tell "crackles" from "no crackles" and confuses "plan of care" with "withdrawal of care" (A Multimodal Transformer). Such a model can look accurate on average while responding to vocabulary, writing style or documentation density rather than to what the note actually asserts about the patient.
We propose to fix this with counterfactual data augmentation, meaning training on "minimal pairs": copies of real examples that differ from the original in exactly one controlled way. For each real note we generate two kinds of edited twin. The first kind changes only something clinically irrelevant: abbreviation style, section order, boilerplate, or demographic descriptors. The second kind changes only the assertion status of one finding, for example present to absent, current to historical, or definite to possible. The model is trained to give the same output for the first kind and to register the change for the second. The structured table is the multimodal ingredient. Assertion edits are allowed only on findings that have no counterpart in the table, so the edited patient remains internally coherent. Findings that do have table counterparts become a separate probe of which source the model trusts when note and table disagree.
Core Claim
The claim is that end-to-end note–table fusion models trained only on observed data do not use assertion-level note content. They are also measurably sensitive to non-clinical note features. Training on a mix of "should not matter" and "should matter" note edits, with the table held fixed, should correct both problems. We also make a falsifiable prediction about where accuracy gains should appear: on nursing and radiology notes, which carry information absent from the tables, and not on physician notes, which largely restate them. On the Importance of Clinical Notes reports exactly this split in note usefulness.
The idea differs from existing counterfactual work in two ways. First, it is two-sided: it enforces invariance to some edits and sensitivity to others, where prior work enforces invariance only. Second, the edits are made to text and checked against a paired table.
Relation to Prior Work
Three groups of papers shape this proposal.
Papers that supply the invariance technique. Word-swapping augmentation from NLP fairness work swaps paired terms such as he and she so a model cannot use them as shortcuts (Zhao et al.; Gupta et al.). A later study shows that dictionary swaps break grammar and that LLM rewriting with human checking preserves it (Nahurna). We adopt LLM-driven rewriting for that reason. Counterfactual Reasoning for Fair Clinical Risk Prediction supplies the training penalty. It generates demographic "twins" of patients over structured codes using a variational autoencoder, a generator that learns a compressed latent description of each patient. It then applies counterfactual logit pairing, a penalty whenever the model's raw scores differ between a patient and their twin. That work ignores text entirely, and it assumes a causal graph that cannot be tested. We apply the same pairing penalty to text edits whose irrelevance holds by construction, so no causal graph is needed.
Papers that motivate the gap. The Multimodal Transformer's own admission of negation failure is the direct motivation. The note-type ablation in On the Importance of Clinical Notes tells us where note content should matter. PheME shows that keyword-filtered notes already raise phenotype accuracy substantially. That result is consistent with fusion models using notes at the keyword level and never needing assertions.
Papers that constrain the design. CACHE uses "counterfactual" to mean masking subsets of billing codes for explanation. It does not generate data, so it does not overlap with our approach. EHR-ReasonCon finds that frontier LLMs verify note–table consistency with F1 only "in the 50s". This warns us not to trust an LLM alone to certify that an edit is coherent with the table. Our coherence rule is therefore structural: only findings with no table counterpart may be edited. The LLM verifier serves only as a secondary filter. Finally, Counterfactual Augmentation for Multimodal Learning Under Presentation Bias generates counterfactual labels in recommendation data. We instead generate counterfactual inputs and never invent outcome labels.
Research Program
Building the edited notes. We start from MIMIC-III ICU stays with notes from the first 48 hours, following the cohort setup of A Multimodal Transformer. A clinical concept tagger marks candidate findings. Each finding is linked to the table's variables, and a finding is flagged as "table-covered" if a corresponding lab, vital sign or code exists. Examples are fever versus temperature, or intubation versus ventilation records.
An LLM rewriter then produces the two kinds of edit.
- Nuisance edits cover abbreviation expansion and contraction, section reordering in the style of the section taxonomy of MedSecId, boilerplate insertion, and swapping race and sex descriptors. Demographic swaps are made only when the table's demographic fields are swapped in step and the patient has no sex-specific codes.
- Assertion edits flip one uncovered finding's negation, time frame or certainty, with minimal surrounding change.
Each edit is filtered twice. A string-difference check confirms that only the target span and grammatically required words changed. The LLM verifier then checks the edit against the table. A random subset is audited by clinicians, who judge nuisance edits "clinically equivalent" and assertion edits "correct and minimal."
The training objective. The objective has three terms.
$$ L = L_{\text{task}} + \lambda_{\text{inv}} \, \mathbb{E}\,\lVert f(x) - f(x^{\text{nuis}}) \rVert^2 + \lambda_{\text{sens}} \, \mathbb{E}\, \ell_{\text{assert}}\big(h(z(x^{\text{assert}})), a\big) $$
The symbols are defined as follows. $f$ is the fusion model's output score. $x$ is a real note paired with its table. $x^{\text{nuis}}$ and $x^{\text{assert}}$ are its nuisance-edited and assertion-edited twins. $z$ is the text branch's internal representation, meaning the vector the text encoder produces before fusion. $h$ is a small head that reads assertion status from that representation. $a$ is the assertion label, which is known because we made the edit. $\ell_{\text{assert}}$ is a cross-entropy loss.
The second term is counterfactual logit pairing on nuisance twins. The third term forces the fused text representation to carry the edited assertion. We deliberately impose no rule about which way the outcome score should move under an assertion edit, because that direction is clinically uncertain. Separately, the edited notes serve as a held-out stress test. Clinicians label a subset of assertion edits by the expected direction of risk change, for example "intubated" versus "not intubated" for mortality risk.
Baselines
The naive reference point is the MBERT-based Multimodal Transformer trained on observed data only, using its published cohort split of 14,068 / 3,086 / 3,107 stays. MBERT is Clinical BERT further fine-tuned on MIMIC notes. For in-hospital mortality it reports AUCPR 0.538, AUCROC 0.877 and F1 0.490 (A Multimodal Transformer). We will reproduce it before adding anything.
The strongest existing fusion competitor is the Cross-Modal Transformer of On the Importance of Clinical Notes. In this model, table time series query Clinical BERT note embeddings. It reports decompensation AUPRC 39.7, against 31.7 for an EHR-only Transformer. We will train our objective on top of both architectures with identical cohorts, encoders, number of training steps and hyperparameter search budget. Every model, including baselines, receives the same number of extra training examples. For baselines those extras are unedited duplicates or plain paraphrases, so gains cannot come from more data.
The pipeline competitor is the alternative we consider most threatening. It first extracts assertion-labeled concepts with a rule-based negation and temporality tagger, then feeds them to the fusion model as extra structured features alongside the raw note. This is the CheXbert-style approach of turning text into labels and then modeling on the labels (CheXbert).
Ablations isolating the key ingredient include invariance-only, sensitivity-only, and random-edit augmentation. The random-edit condition applies the same number of edits but sends them to both terms without distinguishing "should matter" from "should not." We also include a demographic-masking baseline that deletes descriptors instead of swapping them.
What counts as success. We beat the baselines if two conditions hold. First, on the counterfactual test suite, nuisance-edit score changes fall and clinician-labeled assertion-edit direction accuracy rises, both by margins that exceed seed-to-seed variation. Second, task AUCPR on mortality and decompensation does not drop by more than the fairness-utility cost observed for logit pairing on structured data. Counterfactual Reasoning for Fair Clinical Risk Prediction reports prolonged-length-of-stay AUC-ROC falling from 0.851 to 0.843 under strong pairing.
Why not just…?
Why not just mask demographic words and fields? Masking removes explicit tokens but leaves proxies intact. Counterfactual Reasoning for Fair Clinical Risk Prediction argues that care patterns and code frequencies redundantly encode demographics. RadFusion notes that even images carry race information. More importantly, masking does nothing for negation, which is the main target here. The masking baseline is run directly, and we expect it to leave the assertion stress-test scores unchanged.
Why not just extract negated concepts with a tagger and add them as features? This is a real threat. If the pipeline matches our method on both the stress test and task accuracy, the honest conclusion is that end-to-end counterfactual training is unnecessary, and we will report that. We still expect an advantage for two reasons. Taggers cover a fixed vocabulary and fail on clinical shorthand (Sontag et al.). A pipeline also gives no invariance to style or demographic cues in the raw note that the model still reads. The pipeline-competitor comparison, broken down by tagger-covered and tagger-uncovered findings, settles this.
Why not just drop the notes and use tables? The file's evidence says the notes carry real signal. Decompensation AUPRC rises from 31.7 to 39.7 when notes are added (On the Importance of Clinical Notes), and the MIMIC-III mortality work shows a 0.460 to 0.538 AUCPR gain (A Multimodal Transformer). The EHR-only model is included as a floor.
Why not just ask a frontier LLM to read note and table together? EHR-ReasonCon finds F1 only "in the 50s" on note–table reasoning. Benchmarking Foundation Models finds zero-shot vision-language models weaker on length of stay. We will nonetheless run a zero-shot LLM on the same stress test, because it may read negation well even if its risk calibration is poor.
Decisive Checks
The first check is the note-type prediction. Assertion-edit training should raise AUCPR more for nursing and radiology notes than for physician notes. Uniform gains would suggest the benefit is generic regularization rather than improved reading.
The second check guards against circularity, since the stress test could reward models for learning the editor's quirks. Test-time edits come from a different LLM than training edits, plus a small set of clinician-written minimal pairs, and results must hold on the clinician set.
The third check is the conflict probe on table-covered findings. For example, a note says "afebrile" while the table shows a high temperature. This probe reports whether training shifts the model toward trusting the table. That behavior is desirable given the error rates in notes, and it connects to the modality-imbalance concern raised by CareBench.
Finally, we report true positive rate gaps across race and sex subgroups before and after training. This speaks to the disagreement between CareBench and Benchmarking Foundation Models over whether fusion widens disparities. For notes-plus-table models, no reported subgroup result was found on Lacuna.
References
- A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction
- On the Importance of Clinical Notes in Multi-modal Learning for EHR Data
- Gender Bias in Contextualized Word Embeddings
- Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal
- Gender Swapping as a Data Augmentation Technique: Developing Gender-Balanced Datasets for Ukrainian Language Processing
- Counterfactual Reasoning for Fair Clinical Risk Prediction
- PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data
- Counterfactual and Factual Reasoning over Hypergraphs for Interpretable Clinical Predictions on EHR
- Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records
- Counterfactual Augmentation for Multimodal Learning Under Presentation Bias
- A New Public Corpus for Clinical Section Identification: MedSecId
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
- RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
- Benchmarking Foundation Models with Multimodal Public Electronic Health Records
- When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion