When an intensive care patient is suspected of a bacterial infection, clinicians must choose an antibiotic one to three days before the lab reports which drugs the organism actually resists. Models that guess resistance from the electronic health record already work well when a patient has a documented history of resistant cultures. They work much less well for newly admitted or transferred patients, who arrive with no microbiology history at that hospital. We propose that the free-text notes of these "cold-start" patients often hold the missing history in plain words: "MRSA at outside hospital," "recent ESBL UTI," "nursing home resident," "on cipro since last week." The plan is to pull a small set of clinician-specified resistance risk factors out of the notes as discrete facts, each tied to the sentence that supports it. These facts become extra columns for a gradient-boosted tree model, an ensemble of small decision trees in which each new tree corrects the errors of the earlier ones (XGBoost is the standard implementation). We then measure whether the gain from text is concentrated in exactly the patients where structured history is absent.
The second half of the contribution is methodological, and it may matter more. Notes paired with cultures can mention the culture result itself ("cultures grew Pseudomonas, resistant to..."). Any model that reads such notes is partly reading the answer. We will enforce a strict rule that only text charted before the culture order may be used, and we will measure how much published text-based performance depends on breaking that rule.
Core Claim
Text adds useful, auditable signal for antibiotic-resistance prediction mainly through resistance history and exposure that the structured record at the current institution does not capture. Three things follow from this claim. First, extracting a handful of named facts from pre-order notes and adding them to a structured-feature XGBoost should improve precision on resistant cases. Second, the improvement should be markedly larger for patients without prior in-system resistance results than for patients with them. Third, the named facts should match generic sentence embeddings in accuracy while staying readable and checkable at the bedside. If embeddings beat the named facts by a clear margin, the claim is wrong about where text's value lives, and we will report that.
Relation to Prior Work
The closest work is Predicting Antibiotic Resistance Patterns Using Sentence-BERT. It encodes MIMIC-III notes with Sentence-BERT and feeds the vectors to XGBoost, reaching an average AUC of 0.78 over seven antibiotics. It differs from our proposal in three ways. It uses text alone without structured features. It splits on unreadable embedding dimensions. It does not describe a cutoff that excludes notes written after the culture result was known. Our proposal keeps XGBoost but changes what it reads and when.
Affordable and real-time antimicrobial resistance prediction from multimodal electronic health records motivates the text modality. It reports that fusing notes with vitals and labs in neural fusion networks improves AUROC by roughly 10% over single-modality baselines on MIMIC-IV. It also shows the cost of rare outcomes: gentamicin AUROC is 0.7565 but the area under the precision-recall curve (AUPR) is only 0.1825. We do not claim a tree model removes that gap. Low AUPR at a low base rate is partly a property of the task, so we report AUPR against its base rate rather than in isolation.
Two structured-only studies set the bar text must clear and identify our target subgroup. Predicting Antimicrobial Resistance in the Intensive Care Unit reaches an ensemble AUC of 0.8946 on eICU, against 0.8601 for the hospital's summary susceptibility table (the antibiogram). It relies heavily on a prior-resistance flag, which is missing for new patients. Machine Learning for Antibiotic Stewardship in the Treatment of Stapholycoccus Bacterial Infections uses LightGBM on 4,754 structured variables and reaches AUROCs of 0.87–0.93 for Staphylococcus while ignoring notes. Its high-specificity threshold analysis supplies our stewardship metric.
For extraction we reuse methods rather than invent them. Leveraging transformers and large language models with antimicrobial prescribing data shows that fine-tuned Bio+Clinical BERT extracts infection source and diagnostic uncertainty at F1 0.97, against 0.84–0.86 for n-gram XGBoost and 0.71–0.74 for regular expressions. It also finds that text recovers 31% more specific infection sources than billing codes. Its scope was short prescribing strings and source classification, not susceptibility. Self-Verification Improves Few-Shot Clinical Information Extraction supplies the evidence-grounding step: an extraction must cite its supporting span or be pruned. We use it to label training data, not at inference. Imbalance-XGBoost supplies weighted and focal objectives for XGBoost. On its 42:1 ozone benchmark, weighting alone raises F1 from 0.144 to 0.268. That result is why we treat simple class weighting, not focal loss, as the default.
Research Program
The cohort is MIMIC-III (MIMIC-III, a freely accessible critical care database). It has timestamped nursing and physician notes, which the strict pre-order rule requires. Each example is one culture with susceptibility results for one antibiotic, labeled resistant (including intermediate) or susceptible, following the Sentence-BERT paper's convention so the two studies can be compared. The prediction time is the culture order timestamp. Inputs are restricted to structured data recorded before that time, notes charted before that time, and discharge summaries from earlier admissions.
We build the structured feature set in the style of the stewardship and ICU studies: demographics, recent vitals and labs, devices, prior in-hospital antibiotic administrations, and prior in-system resistance flags per drug class. Each patient-antibiotic pair is then tagged "warm" if any prior in-system susceptibility result exists and "cold-start" otherwise.
The text component is a fixed schema of about fifteen facts chosen with infectious-disease clinicians. Each fact has a status: affirmed, negated, historical, or uncertain. The schema covers:
- prior resistant organisms and where they were documented, including outside facilities;
- recent antibiotic exposure outside the record;
- long-term care or recent hospitalization elsewhere;
- dialysis and indwelling devices mentioned only in narrative;
- immunosuppression;
- suspected infection source;
- explicit diagnostic uncertainty.
Clinicians annotate about 1,000 pre-order notes against this schema to create an extraction test set. A large instruction-tuned language model with the self-verification pipeline labels a larger training pool. A Bio+Clinical BERT tagger is then fine-tuned on that pool for fast inference, since the Oxford study reports BERT at 82 seconds per 10,000 strings against 1.2 seconds for XGBoost.
The downstream model is XGBoost with class-weighted log-loss, one model per antibiotic. Focal loss is tested only as an ablation. Every alert surfaces the extracted facts that drove it, using SHAP attributions over named columns, together with their supporting sentences.
Baselines
The naive reference is the institutional antibiogram: each organism-class and drug pair is assigned its historical resistance rate from the training years. Predicting Antimicrobial Resistance in the Intensive Care Unit reports this baseline at AUC 0.8601 on eICU. That number comes from a different database, so we recompute it on our split. Any model that fails to beat it on cold-start patients is not useful.
The strongest structured competitor is XGBoost on structured features alone, tuned exactly as the full model: same splits, same 100-trial hyperparameter search, same class weighting. The closest comparison in the literature is the 0.87–0.93 AUROC of the Staphylococcus stewardship models, but that is one genus with LightGBM. The proposal beats this baseline if adding text facts raises AUPR on cold-start patients by a margin whose bootstrap 95% interval excludes zero. A smaller or null gain on warm patients is expected and consistent with the claim.
The strongest text competitor reimplements Sentence-BERT plus XGBoost, which reports an average AUC of 0.78 and 0.81 on meropenem. We run it twice, once with all paired notes as in the original and once under our pre-order rule. We also run a stronger variant that concatenates the embedding, reduced by principal components to 64 dimensions, with the structured features. That variant is the real threat to the "named facts" claim.
The neural fusion competitor is MultiModal Infomax as described by Hardan et al., retrained on our cohort and cutoff. Their reported figures are AUROC 0.7565 and AUPR 0.1825 for gentamicin on MIMIC-IV, but those come from a different cohort and timing.
Three ablations isolate the key ingredient:
- Statuses removed: keep the facts but drop affirmed, negated, historical and uncertain, so any mention counts as affirmed. This tests whether negation and temporality matter.
- Regular-expression extraction: replace the BERT tagger with keyword rules for the same schema.
- Single-category set: keep only the outside-history facts. This tests whether the gain lives where we claim.
Why not just…?
Why not just use structured data, which already performs well? On warm patients it may well suffice, and we expect text to add little there. The open question is cold-start patients, where the prior-resistance flag that Wang et al. depend on is absent by definition. The structured-only baseline, stratified by warm and cold, settles this directly. If text adds nothing even for cold-start patients, the proposal fails.
Why not just concatenate sentence embeddings with structured features? This is the most serious alternative, and it may match our accuracy. Our bet is that the named facts lose little signal while gaining an auditable trail and robustness across eras, because a fixed schema should drift less than embedding geometry when writing habits change. We test this on a temporal split: train on 2001–2008 and test on 2009–2012, where Wang et al. already observed decay. If embeddings win clearly in both accuracy and temporal robustness, we will report that text's value is diffuse rather than concentrated in history facts, and drop the core claim.
Why not just fix imbalance with focal loss or end-to-end neural fusion? The Imbalance-XGBoost results show most of the gain on skewed data comes from simple weighting (0.144 to 0.268 F1). We therefore expect focal loss to add little beyond weighting and include it only as an ablation. Neural fusion is retrained under identical inputs. Its documented low AUPR suggests no advantage at this sample size, but if it wins we report it.
Why not just keyword-match "MRSA" or "ESBL"? The prescribing-indication study found rules at F1 0.71–0.74, and CheXbert found rule labelers at 0.743 against 0.798 for BERT. Resistance history is full of negated and historical mentions ("MRSA screen negative," "prior VRE, cleared"). The regex ablation tells us whether that extraction gap carries through to susceptibility prediction or washes out in the trees.
Decisive Checks
The first check is the leakage audit. We compute the Sentence-BERT pipeline's AUC with all paired notes and again with pre-order notes only. A drop of more than a few points would mean part of the reported text signal is the answer leaking in. That finding stands on its own regardless of the other results.
The second check is the subgroup interaction. The AUPR gain from text facts must be larger for cold-start than warm patients, tested with a bootstrap interval on the difference.
The third check compares named facts against embeddings, in-era and on the temporal split.
The fourth check is clinical utility. At a threshold that keeps sensitivity for resistance at 0.95, we report the fraction of patients for whom a narrower agent would have been flagged safe. This follows the high-specificity analysis of the stewardship study. Alongside it we report calibration error and extraction F1 against the clinician-annotated set.
External validation across institutions is not possible with this file's resources. eICU lacks notes, and MIMIC-IV comes from the same hospital. We state this limitation plainly and treat the MIMIC-III to MIMIC-IV era transfer only as a partial, same-site substitute.
References
- Predicting Antibiotic Resistance Patterns Using Sentence-BERT: A Machine Learning Approach
- Affordable and real-time antimicrobial resistance prediction from multimodal electronic health records
- Predicting Antimicrobial Resistance in the Intensive Care Unit
- Machine Learning for Antibiotic Stewardship in the Treatment of Stapholycoccus Bacterial Infections
- Leveraging transformers and large language models with antimicrobial prescribing data to predict sources of infection for electronic health record studies
- Self-verification improves few-shot clinical information extraction
- Imbalance-XGBoost: leveraging weighted and focal losses for binary label-imbalanced classification with XGBoost
- MIMIC-III, a freely accessible critical care database
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT