Clinicians write notes the way everyone writes: they record what is surprising and skip what "everyone knows." A note may say "worsening dyspnea" but not mention the heart failure diagnosed two admissions ago, or the creatinine that has been high for years. That history sits in the hospital's structured tables of billing codes and lab results, or in another hospital's system entirely. A transformer fine-tuned on these notes learns to treat what is written as the whole patient. It then fails on rare outcomes, and it fails again when moved to a setting where different things go unwritten. We propose a cheap source of supervision for this gap. For each admission, the hospital's own structured records reveal which clinically relevant facts were true of the patient but never stated in the note. We fine-tune the text model to predict those unstated facts alongside its main task.
This targets a failure that the general commonsense literature has named but not studied in medicine. In the clinical setting, the omitted background is not universal physics. It is patient-specific, and it depends on local documentation habits. The same records that expose the omissions cannot legally be pooled across institutions. The method therefore has to run inside each data silo, with no raw data leaving it, and its behaviour has to be inspectable by oversight bodies that do not share a common rulebook.
Core Claim
The claim is that making a clinical text encoder predict the facts its note omits, using labels computed automatically from the same site's structured records, does three things. It improves rare-outcome prediction from admission notes. It makes predictions more stable when documentation conventions shift. It does both better than the obvious alternatives: feeding the structured data in directly, reweighting the loss, or asking a large language model to write out the background.
Two properties of the labels matter. They are gap labels: facts present in the tables and absent from the text, not all structured facts. They are also privileged: they are used only during training, so the model needs no structured data at prediction time. This is the situation created by fragmented care, where a patient's history often lives in another system.
Concretely, let $x$ be an admission note, $y$ the outcome label, and $U(x)$ the set of structured facts, such as prior diagnosis codes or abnormal lab flags recorded before the admission time, that a concept matcher fails to find stated in $x$. The fine-tuning objective is
$$\mathcal{L} = \mathcal{L}_{\text{task}}(x, y) + \lambda \sum_{f \in F} \mathrm{BCE}\big(g_f(x),\, \mathbb{1}[f \in U(x)]\big).$$
Here $F$ is a fixed vocabulary of candidate facts, $g_f$ is a small head on the encoder that predicts whether fact $f$ holds but is unstated, BCE is binary cross-entropy, and $\lambda$ weights the auxiliary term.
Where This Sits in Prior Work
Three papers motivate the gap.
- Revisiting Clinical Outcome Prediction for MIMIC-IV benchmarks biomedical encoders on outcome prediction from admission notes. It finds PubMedBERT best and reports a severe collapse in precision-recall on rare diagnoses. It treats the note as a complete record, and we do not.
- Automated Medical Coding on MIMIC-III and MIMIC-IV shows that the tail of rare labels, not document length, drives coding errors. It also shows that stripping numbers discards evidence clinicians rely on implicitly. We take both lessons as design constraints: we keep numerals, and we target the tail.
- Predicting onset of complications from diabetes and Nonstationary Gaussian Process Regression for Evaluating Repeated Clinical Laboratory Tests show that patients move between systems. As a result, missing data often means "recorded elsewhere" rather than "normal." This is exactly the ambiguity our gap labels expose.
Three lines of work supply the method or come closest to it.
- Predicting Clinical Trial Results by Implicit Evidence Integration pre-trains BioBERT on 12 million comparative sentences mined from the literature, so the model absorbs implicit trial logic. We borrow its core move, which is to harvest implicit knowledge automatically at scale. Our source is different: the discrepancy between a patient's note and that patient's own record, rather than published text.
- IAG has a generator write out unstated premises before answering.
- RACO retrieves such premises from a 20-million-document commonsense corpus.
Both operate on general-domain benchmarks. Both add inference-time machinery, and IAG's authors flag hallucinated premises as a weakness. Our model instead internalizes the premises through training, and its premise head is checked against real records.
Two further papers constrain the design.
- Looking for Out-of-Distribution Environments in Critical Care finds that heavily parameterized models overfit local hospital quirks under shift. We therefore must show the auxiliary task helps rather than adding more capacity to memorize a site. We also include a smaller-model comparison.
- Federated Learning in Multi-Center Critical Care Research shows that federated averaging across 208 eICU hospitals works for a small tabular network, with long local training and small batches helping. It never tries transformers on text. That is our secondary setting.
On the regulatory side, Regulating the Safety of Health-Related Artificial Intelligence argues two points. Clinicians cannot audit opaque inferences in real time. Adaptive models also collide with one-time premarket licensing. When Less Regulation Means More Complexity documents how 27 divergent national liability regimes raise compliance burdens.
We use these papers as a constraint, not as a modelling target. Gap labels can be computed entirely inside a site. The premise head's output, "this model assumed the patient has chronic kidney disease although the note did not say so," is a checkable artifact that an auditor under any regime can compare against the record. We do not claim to measure regulatory outcomes.
Research Program
Stage 1: build the gap labels. On MIMIC-IV we take the admission-time notes used in the clinical outcome prediction tasks. For each admission we gather candidate facts from structured data dated before that admission's time: diagnosis codes from earlier admissions and flagged abnormal labs. We map codes and labs to a fixed vocabulary of a few hundred clinically common conditions. A concept matcher with negation handling then decides whether each fact is stated in the note. Numerals are preserved, following Edin et al.
Matcher error puts a ceiling on everything downstream. Clinicians will therefore manually audit a sample of note–fact pairs to estimate how often "unstated" is wrong.
Stage 2: fine-tune with the objective above. We use PubMedBERT, the strongest encoder in the MIMIC-IV benchmark, and train on the diagnosis, mortality, and routing tasks. The auxiliary heads are discarded at prediction time for the task score. They are retained as the audit output.
Stage 3: evaluate under shifts that mimic fragmentation. MIMIC-IV is a single hospital, so true cross-institution text shift is not available in the resources we have. This is the main limitation. We approximate it in three ways.
- Coding-regime split. We use the ICD-9 versus ICD-10 era split used by Edin et al. We train on the earlier era and test on the later, when documentation and billing conventions changed.
- Care-setting split. We train on the intensive care unit (ICU) and test on the emergency department (ED), and the reverse.
- Pseudo-silos. We partition by care unit and run federated averaging with the local-epoch and batch-size settings Mehrjou et al. found favourable. This tests whether gap supervision survives when each silo computes its own labels.
Baselines
Naive reference. PubMedBERT fine-tuned on the task alone, reproducing the published MIMIC-IV setup. On ED diagnosis prediction, Revisiting Clinical Outcome Prediction for MIMIC-IV reports an AUROC of 86.86% and a PR-AUC of 17.24% for PubMedBERT. The other encoders reach roughly 14–15% PR-AUC. For patient routing, the paper describes high AUROC and weak PR-AUC but gives no number that we have. Before any comparison, our reproduction must land near the reported figures.
Strongest imbalance-handling competitor. The same encoder with inverse-frequency class-weighted loss, following xiacui at SemEval-2025 Task 11, plus per-class threshold tuning on validation data, following Edin et al. This is the standard fix for rare-class collapse, so it is the natural competitor on our headline metric.
Structured-input upper reference. The same encoder with the prior-admission codes and labs serialized into the input text at both training and test time. We run it twice: with the structured data fully present, and with it randomly removed at test time for a fraction of patients. The removal simulates histories lost to care elsewhere.
Premise-generation competitor. An IAG-style pipeline. A large language model writes the likely unstated background for each note, and that text is appended before fine-tuning.
Isolating ablation. An auxiliary head trained on all prior structured facts, whether or not the note states them. It uses the same heads, the same $\lambda$ search, and the same data. It isolates whether the benefit comes from the gap specifically or merely from multitask learning with extra labels.
Parameter-count control. A smaller encoder, answering the Spathis–Hyland concern about overparameterization under shift.
Fair-comparison conditions. - All systems share splits, tokenizer, and training compute. - Each system gets an equal hyperparameter search budget, with $\lambda$ and the class-weight scheme counted in that budget. - Every system receives per-class threshold tuning. - Results are averaged over five seeds.
What counts as winning. On PR-AUC for the rarer half of labels, gap supervision must beat the weighted-loss baseline by more than seed variance, both in-distribution and under the era and care-setting shifts. It must also beat the all-facts ablation.
Why not just…?
Why not just give the model the structured data directly? This is the most serious alternative. When a patient's full history is available at prediction time, we expect direct input to match or beat us, and the structured-input reference is designed to show this honestly. Our claim is narrower. Under the fragmentation documented by Thomas et al. and Lasko, histories are often missing or held in another system. A model trained to lean on serialized history should then degrade more sharply when that history disappears than a model that learned to read implications from the text. The removal experiment settles this directly. If direct input degrades gracefully, the proposal's contribution shrinks to the audit output, and we will report that.
Why not just reweight the loss and tune thresholds? Xia Cui found that weighted loss left RoBERTa essentially unchanged and that the gains concentrated in frequent classes, with minority classes stagnant. Edin et al. locate the coding bottleneck in label sparsity, which reweighting does not add information to fix. Gap supervision does add information: it supplies rare-condition evidence the note never states. Stratifying results by label frequency will show whether our gains land in the tail, where reweighting's do not.
Why not just have a large language model write out the missing background? IAG shows that generated premises help on commonsense benchmarks, and its authors name hallucinated premises as the open problem. In medicine, a hallucinated comorbidity is a safety defect of exactly the kind Da Silva et al. describe. We measure it rather than assume it. We score the generated premises against the structured record, and we compare their precision with that of our premise head. We also compare downstream accuracy. The generation pipeline further adds per-note inference cost, which we report.
Why not just treat this as ordinary multitask learning? The all-facts ablation answers this directly. If predicting all prior facts helps as much as predicting only the unstated ones, the "implicit gap" framing is unnecessary, and we will say so.
Proposed Experiments and Decisive Checks
Main comparison. We train all baselines on the MIMIC-IV admission-note tasks. The metrics are AUROC and PR-AUC for diagnosis prediction, stratified into frequency bins, and PR-AUC for mortality and patient routing. Each result is reported in-distribution, across the ICD-9→ICD-10 era shift, and across the ICU↔ED shift.
Omission-sensitivity test. For held-out admissions, we delete note sentences that state a fact confirmed by the structured record. We then measure how much each model's outcome probability moves. A model that has learned the omitted background should move less, because it can still infer the fact. The task-only baseline should move more. This is the most direct test of the mechanism. It is decisive if the gap-trained model is markedly more stable while also scoring higher on the main metrics.
Premise-head quality. We score the premise head's predictions against the record and against the clinician-audited sample. This establishes whether its "assumed but unstated" outputs are accurate enough to serve as an audit trail.
Federated check. We rerun the main comparison under federated averaging across pseudo-silos. The question is whether locally computed gap labels keep their advantage when no silo sees another's data.
Failure conditions. The proposal fails if any of the following holds:
- the tail PR-AUC gain over weighted loss is within seed noise;
- the all-facts ablation matches it;
- the gain vanishes under the era shift.
Any of these would indicate that the method is extra labels, not a remedy for what notes leave out.
References
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- Predicting onset of complications from diabetes: a graph based approach
- Nonstationary Gaussian Process Regression for Evaluating Repeated Clinical Laboratory Tests
- Predicting Clinical Trial Results by Implicit Evidence Integration
- IAG: Induction-Augmented Generation Framework for Answering Reasoning Questions
- Retrieval Augmentation for Commonsense Reasoning: A Unified Approach
- Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
- Federated Learning in Multi-Center Critical Care Research: A Systematic Case Study using the eICU Database
- Regulating the Safety of Health-Related Artificial Intelligence
- When Less Regulation Means More Complexity: The EU AI Liability Directive Withdrawal and Its Impact on European Technological Competitiveness
- xiacui at SemEval-2025 Task 11: Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss