Separating What Patients Report From How Clinicians Frame It in MIMIC-IV Notes

When a clinician writes "patient claims chest pain since morning" instead of "patient reports chest pain since morning," the medical content is the same. The note now also carries a judgment about whether the patient is believable. Language models fine-tuned on clinical notes read both. If credibility framing is used more often for some patient groups than others, and if the model's predictions change when only the framing changes, then a text-based clinical predictor can pass the clinician's skepticism on as a demographic disparity, even when no demographic variable is ever fed to it. This note proposes to test both conditions on MIMIC-IV, a large public database of intensive-care and emergency-department records from one Boston hospital. It also proposes a fix. A rule-based linguistic layer separates the reported content of a sentence from the framing applied to the person reporting it, and the neural model is trained to keep the first and ignore the second.

The opportunity exists because two findings have not yet been connected. Patel and Pavlick showed on Wikipedia text that verbs like "claimed" versus "stated" push generative models toward polarized continuations, and that this sensitivity does not shrink with model size "Was it "stated" or was it "claimed"?. Separately, MIMIC-IV text benchmarks report demographic performance gaps without explaining them. For example, patient routing is systematically worse for male patients in Revisiting Clinical Outcome Prediction for MIMIC-IV. Nobody has measured framing in MIMIC-IV notes, asked whether it varies by patient group, or tested whether outcome models depend on it.

Core Claim

We make three claims, each testable on its own. First, after matching patients on age, sex, chief complaint and triage acuity, credibility framing of patient testimony occurs at different rates across racial and sex groups in MIMIC-IV emergency and discharge notes. Second, a fine-tuned clinical encoder changes its diagnosis and routing predictions when only the credibility framing in a note is rewritten. Third, a hybrid model removes most of that sensitivity, and narrows subgroup gaps, without losing precision on rare diagnoses.

The key design choice is that the linguistic features are not appended as extra inputs. Concatenation is the integration style that Hybrid Models for Sentence Readability Assessment found to underperform. Instead, the features define the two things the model must keep apart: what it must preserve and what it must ignore.

The linguistic layer matters because of a distinction that a word list cannot draw. Hedges about findings are clinically informative, for example "possible pneumonia" or "cannot rule out PE." Skepticism about the patient as a witness carries a credibility judgment, for example "claims," "insists," "reportedly," "per patient," "allegedly," or "drug-seeking." The same word, such as "reportedly," can attach to either. Deciding which one it attaches to requires parsing who the subject of the reporting verb is and what the scope of the hedge is. That is the parser's job.

Relation to Prior Work

The closest work falls into three groups, and none does what we propose.

The first group audits MIMIC-IV but never touches note language. MIMIC-IF audits five deep sequence models for fairness on structured MIMIC-IV variables. It finds that higher-mortality subgroups get lower AUC and that ethnicity is weighted unevenly, but it uses no notes and tests no mitigation. Racial differences in laboratory testing uses a 1:1 exact-matched design on the MIMIC-IV emergency split. It shows that White patients receive more blood counts and metabolic panels and Black patients more troponin tests, and argues that models will read missing labs as missing disease. We reuse its matched-cohort design for our measurement study and move it from lab orders to narrative language. Both papers motivate the proposal.

The second group studies shortcuts in clinical text but stops at prompting. Spurious Correlations and Beyond shows that Llama models extracting substance use from MIMIC-III social histories treat alcohol and smoking mentions as shortcuts, with a 66.21% false-positive rate for drug use when alcohol is mentioned. False-positive rates are higher for men. Chain-of-thought prompting and fine-tuning lower aggregate errors, but the gender gap persists. That paper motivates the proposal and also constrains it: prompt-level fixes do not close subgroup gaps, so we work at the level of the training objective. It also uses MIMIC-III and an extraction task, while we use MIMIC-IV outcome prediction.

The third group supplies the method. Evaluating the Utility of Hand-crafted Features in Sequence Labelling attaches an auxiliary autoencoder that reconstructs part-of-speech, word-shape and gazetteer features from a recurrent tagger's hidden states. It reports 91.89 F1 on CoNLL-2003 named-entity recognition and parity with its baseline using 60% of the training data. We borrow this "reconstruct the symbolic features" objective to keep clinical content in the representation. FairerOPTH Fairer AI in ophthalmology grounds retinal disease predictions in 67 explicit lesion features and narrows sex and age gaps without adversarially removing demographics. We transfer its central idea, grounding predictions in features causally tied to the outcome, from images to text.

The same paper also gives a warning. Its own dataset came from a single centre, and so is ours. Any fairness gain we show is a single-institution result.

Research Program

The work has three parts.

The linguistic layer. We build a rule-based attribution and scope parser over MIMIC-IV notes. It combines a reporting-verb and hedge lexicon, negation detection, and dependency scope rules. It assigns each epistemic marker to one of two classes. "Patient-testimony framing" means skepticism or distancing aimed at the patient's account. "Finding uncertainty" means clinician uncertainty about a diagnosis or observation. The parser also emits "content features": the negation-aware symptoms, findings and complaints that the sentence asserts.

Handcrafted extractors are known to degrade on telegraphic clinical text, and dependency tags hurt even on clean news text in the sequence-labelling study. We therefore do not trust the parser blindly. Two clinically trained annotators will label a stratified sample of several hundred marker occurrences, and the parser's precision and recall are reported before anything downstream is run.

Measurement. Following the matched design of the lab-testing study, we match emergency and admission notes on age, sex, chief complaint and acuity. We then compare per-note rates of patient-testimony framing across race groups. Sex comparisons use matching on race instead of sex. Finding-uncertainty rates are reported alongside as a control: if every kind of hedging differs by group, the effect is about documentation style, not credibility.

Sensitivity and mitigation. For every note, the parser produces a framing counterfactual. Patient-testimony markers are swapped to neutral reporting ("claims" becomes "reports," "allegedly" is deleted). Finding-uncertainty markers and content stay fixed. The hybrid model is a PubMedBERT encoder trained with three objectives:

$$\mathcal{L} = \mathcal{L}_{\text{task}}(x) + \lambda_c\, \mathcal{L}_{\text{rec}}\big(c(x) \mid h(x)\big) + \lambda_f\, D\big(p(\cdot \mid x),\, p(\cdot \mid \tilde{x})\big)$$

The symbols are as follows. $x$ is the note and $\tilde{x}$ its framing counterfactual. $\mathcal{L}_{\text{task}}$ is the outcome loss, for multi-label diagnosis or routing. $h(x)$ is the pooled encoder state. $c(x)$ is the parser's content-feature vector, reconstructed from $h(x)$ by a small head. $D$ is a symmetric KL divergence between the model's predictive distributions on the original and counterfactual notes. $\lambda_c$ and $\lambda_f$ are weights tuned on validation data.

The consistency term removes dependence on framing. The reconstruction term stops the encoder from satisfying that constraint by throwing away clinical detail, which is the failure mode we expect from simple masking.

Baselines

The naive reference point is PubMedBERT fine-tuned end-to-end on the COP-IV tasks exactly as in Revisiting Clinical Outcome Prediction for MIMIC-IV. It is also the strongest existing method on these tasks. That paper reports, for emergency-department diagnosis prediction, an AUROC of 86.86% and a PR-AUC of 17.24% for PubMedBERT, while other clinical encoders reached roughly 14–15% PR-AUC. For routing the paper reports that male patients do worse but, in the file available to us, gives no subgroup numbers. For race-stratified performance on these tasks: no reported result found on Lacuna. We will rerun this baseline ourselves on identical splits, so that subgroup and counterfactual numbers come from one implementation. Every model shares the same encoder initialization, data, number of fine-tuning epochs, and hyperparameter search budget.

The second baseline removes explicit demographic mentions (race, sex, age tokens) from notes before fine-tuning. This is the standard "fairness through unawareness" move, and it tests whether the gaps run through overt identity tokens rather than framing.

The third baseline is lexical masking. Every word on the framing lexicon is deleted or replaced with a neutral token, regardless of what it attaches to.

The fourth baseline is lexical counterfactual augmentation. It trains on word-list swaps with the same consistency loss but no parser scoping, so finding hedges get swapped too.

The fifth baseline is a concatenation hybrid. The parser's framing and content counts are appended to the pooled encoder state, which is the integration style reported to fail in the readability work. We also run Liu and Lee's soft-label stacking, which feeds encoder probabilities plus parser features into a gradient-boosted classifier. This tells us whether stacking alone captures the benefit.

Two ablations isolate the key ingredients: the full model without the reconstruction term, and the full model with parser scoping replaced by the lexicon. A structured-only model on MIMIC-IV tables is reported as a reference, not a competitor, because it cannot see framing. MIMIC-IF reports AUROC 0.955 for IMV-LSTM, but on mortality, a different task, so the numbers are not comparable.

The hybrid "beats" the naive baseline if it meets three conditions. Its PR-AUC on diagnosis prediction must be statistically non-inferior, with a pre-registered margin of one point and patient-level bootstrap intervals. Its counterfactual flip rate (the share of notes whose top routing prediction or thresholded diagnosis set changes under framing swap) must be clearly lower. And its matched subgroup gap in routing accuracy and diagnosis PR-AUC must be smaller. It beats lexical masking only if it preserves more performance on notes rich in finding-uncertainty while matching masking's flip rate.

Why not just…?

Why not just mask or delete the skeptical words before training? This is the most serious threat, and it is built into the experiments as the third baseline. The argument against it is specific. A word list cannot tell "reportedly took overdose," which distances the patient's account, from "reportedly hypotensive in ambulance," which reports a second-hand finding. It also cannot tell credibility verbs from diagnostic hedges, which carry real predictive signal for exactly the uncertain, long-tail diagnoses where COP-IV PR-AUC is already low. If masking matches the hybrid on PR-AUC, flip rate and subgroup gaps, then the parser is unnecessary and we will say so. The hybrid's contribution would then reduce to the measurement study. The test that settles this is a comparison stratified by how many finding-uncertainty markers a note contains, where masking should lose most.

Why not just use a larger language model, zero-shot or with a careful prompt? The file gives no reason for optimism. Zero-shot Meditron-7B and Llama-2-13B reach AU-ROC 0.61 and 0.50 on templated MIMIC-IV mortality prediction in Revisiting the MIMIC-IV Benchmark. Patel and Pavlick found that framing sensitivity does not fall with scale. In the substance-use study, chain-of-thought prompting reduced errors but left gender gaps in place. We will still run the framing-counterfactual test on one open instruction-tuned model, to report its flip rate. We do not expect it to compete on the outcome metrics.

Why not just remove demographics, or train adversarially against them? Removing demographic tokens is the second baseline. Our hypothesis is that it fails because framing acts as a proxy for group membership. This is the same mechanism by which missing labs act as a proxy in the lab-testing study. FairerOPTH argues that grounding a model in outcome-relevant features works better than adversarially stripping demographics. Our design follows that argument, but we have not tested it against adversarial removal. If compute allows, we will add an adversarial demographic-removal encoder under the same budget. If it closes subgroup gaps equally well while leaving the flip rate high, then framing invariance and group fairness are separable goals, and we would report that.

Why not just use structured data, where there is no framing? Structured data carries its own documented bias through lab-ordering disparities, and it discards the note text that drives COP-IV routing performance. Röhr et al. report that a detailed referral note helps routing substantially. Structured-only results are therefore reported for context, not as a fix.

Proposed Experiments and Decisive Checks

All experiments use MIMIC-IV notes aligned to the COP-IV emergency-department diagnosis and patient-routing tasks and the intensive-care diagnosis task, with patient-level splits. Race and sex come from MIMIC-IV demographics.

The first decisive check is parser validity. If annotated precision for separating testimony framing from finding uncertainty is poor, everything downstream is uninterpretable, and the project falls back to the lexical baselines. A well-specified annotation guide is then the deliverable.

The second check is the matched measurement. A null result, meaning no difference in testimony-framing rates across groups after matching, is publishable and would change the fairness question. A difference in testimony framing with no difference in finding uncertainty is the result the credibility hypothesis predicts.

The third check is sensitivity. If the PubMedBERT baseline's flip rate under framing swaps is near zero, the mitigation is moot and the work reduces to an audit.

The fourth check is the mitigation comparison described under Baselines. We also break subgroup gaps down by diagnosis frequency, to test whether framing effects concentrate in the ICD-10 tail where Röhr et al. located the PR-AUC collapse.

All results are single-site. The Michigan data used in the lab-testing study is the natural external replication, but it is not public, so we flag cross-institution transfer as untested rather than claiming it.