Several recent papers report that clinical prediction models can move to a new hospital using only a handful of local patients. The recipe differs by paper. Some rewrite a patient's ICU time series as a short text summary produced by a language model. Others attach each record to a medical knowledge graph. In one striking case, 16 patients from a new hospital were enough to approach a model trained on more than 36,000. Every one of these results was measured on retrospective research extracts: cleaned, back-filled copies of the record, pulled long after the patients left. Prospective studies show that live data looks different. At the prediction moment, fields are missing, some "static" values such as body mass index are later overwritten with information from after the encounter, and data feeds sometimes fail outright. At the one hospital where this was measured, that difference in data pipelines, not the passage of time, caused most of the performance lost at deployment. Nobody has asked whether the small-sample portability results hold once the few adaptation patients and the test patients are both drawn from data as it actually existed at prediction time.
We propose to answer that question directly. First, we build a deployment-realistic replay of three public ICU databases, so that every model sees only what a live pipeline would have delivered. Second, we split each model's loss at a new hospital into a site part and a data-availability part. Third, we test one targeted fix. When a language model writes text summaries from incomplete records, it tends to invent the missing lab values, so we constrain it to state only values present in its input and to say explicitly what is absent. The central claim is falsifiable. Either the reported advantage of text-summary representations over simple tabular models at small sample sizes persists under deployment-realistic data, or much of it comes from retrospective completeness. Both answers change what a hospital with 16 labeled patients should deploy.
Why this intersection, and what already exists
The portability claims motivate the work. Record2Vec turns 48 hours of ICU vitals and labs into text with a frozen language model, embeds that text, and trains a light classifier. It wins 10 of 10 cross-site transfers from MIMIC-IV to a private pediatric ICU, and with 16 target samples it approaches a 36,000-sample baseline. The authors attribute this to text acting as a "semantic normalizer" that absorbs missingness and sensor differences. GraphCare attaches patient records to a knowledge graph and reports that 36 patients match baselines trained on 720. MiME encodes the visit, diagnosis and treatment hierarchy and reports a 15% relative PR-AUC gain on its smallest data subsets. PathwayLLM moves a sepsis model from MIMIC-IV to eICU. GraphCare and MiME are single-institution. None of the four evaluates on data restricted to what was available at prediction time.
The prospective literature constrains the design. Mind the Performance Gap introduced a three-snapshot comparison: retrospective development data, live pipeline data, and a later retrospective pull of the same live period. Using it, the authors showed that infrastructure shift cost 0.016 AUROC while temporal shift gained 0.005. They identified retroactively updated features as a source of leakage and recorded pipeline failure on 10 of 356 days. All models are local argues that static external validation is insufficient, citing the Epic Sepsis Model's drop to 0.60–0.64 AUC externally, and recommends local testing with small recent batches. That recommendation is exactly the few-shot regime studied here. Preparing a Clinical Support Model for Silent Mode shows how to run a model on live data without showing clinicians its output, a setup called silent mode, and how to fix an alert budget. Its model was tuned to a positive predictive value of 0.71 at sensitivity 0.40, keeping false alerts to at most two per true alert.
The clinical reasoning gap supplies the failure mechanism we expect. Language models ignore constraints stated in the prompt and fall back on what they learned in pre-training (Wu et al.). PathwayLLM hallucinates specific lab values in about 6% of cases, for example writing "Lactate 4.5" when lactate was never measured. The respiratory guideline auditor of Lu et al. hallucinated in 11 of 20 chart reviews. Deployment-realistic records are sparser than research extracts. A summarizer that fills gaps with plausible numbers is therefore most dangerous in exactly the setting where portability matters. We do not try to close the broader reasoning gap. We use one specific symptom of it, fabricated evidence under missingness, as the thing to measure and suppress.
Core Claim
Small-sample cross-site results for clinical representations are confounded by retrospective completeness. The size of the confound can be measured by evaluating the same models on a timestamp-faithful replay. We decompose the gap between source-site retrospective performance $P(S,\text{ret})$ and target-site deployment-realistic performance $P(T,\text{avail})$ as
$$ P(S,\text{ret}) - P(T,\text{avail}) = \underbrace{[P(S,\text{ret}) - P(T,\text{ret})]}_{\text{site shift}} + \underbrace{[P(T,\text{ret}) - P(T,\text{avail})]}_{\text{availability shift}}, $$
where $P$ is any fixed metric (AUROC, AUPRC, or PPV at the alert budget) and "avail" means inputs restricted to information recorded before the prediction time. Because the split depends on the order in which the two shifts are applied, we also report the reverse order.
We make two hypotheses. The first is that availability shift is not small relative to site shift at small target sample sizes $N$. With 16 to 36 adaptation patients, a classifier readily latches onto leaked or back-filled fields, the same artifact-fitting documented in a chest X-ray study. The second is that text-summary representations keep their advantage only if the summarizer is forbidden from inventing values.
Research Program
The first component is the replay. For MIMIC-IV, eICU and HiRID, we rebuild each prediction instance using only events whose recording time precedes the prediction time, wherever the database exposes such times. Where it does not, we apply conservative lags. Following Ötles et al., we remove or freeze demographic and anthropometric fields that can be retroactively updated. We simulate feed outages by dropping whole data sources, such as all labs or all notes, for contiguous windows at the observed rate of roughly 10 in 356 days. We also run a stress setting at three times that rate. The replay is an emulation. Its validity is checked in the silent-mode stage below.
The second component is a grounded summarizer. Given the available record, the language model must write a summary in which every stated measurement appears verbatim in its input. It must also include an explicit list of which expected measurements are absent. A deterministic checker compares each number in the summary against the input table. On failure, the summary is regenerated. If regeneration keeps failing, the offending sentence is removed. The grounded summarizer then feeds the same embed-and-classify pipeline as Record2Vec, so any difference is attributable to grounding. The goal is not a better model. It is to learn whether a cheap constraint removes the failure that live sparsity is expected to amplify.
The third component is a learning-curve study across target-site sample sizes of 0, 16, 36, 100 and 500 patients. At each size, we sample adaptation patients from the deployment-realistic target data, repeat the draw 20 times, and report intervals. The tasks are in-hospital mortality and sepsis onset, where eICU gives external numbers to anchor against, plus a longer-horizon deterioration outcome defined as in Nestor et al.
Baselines
The naive reference point is the locked source model applied at the target site with no local data, as the external-validation pipeline requires. We run it for every representation, so that the $N=0$ point exists for all methods. Any adapted method that does not beat its own locked version at $N=16$ has shown nothing.
The strongest competitor on this question is gradient-boosted trees (XGBoost) with explicit missingness indicators, trained on the target site's $N$ patients. We also run a pooled variant trained on source data plus the $N$ target patients. ClinicalBench found that traditional models beat 22 language models on MIMIC prediction tasks. On MIMIC-III length of stay, the best traditional model exceeded 70% macro F1 against roughly 26% for the best-prompted language model. Horng et al. report 0.86 AUC from a linear SVM on vitals and bag-of-words triage notes. Trees also handle missing values natively, which directly threatens the claim that text is needed to absorb missingness. The trees get the same features the summarizer sees, the same replay, and the same hyperparameter search budget, measured in configurations tried and chosen by internal cross-validation on the $N$ target patients only.
The published Record2Vec pipeline, run with its ungrounded summarizer on both retrospective and deployment-realistic inputs, is the representation-transfer baseline. For eICU sepsis, the external reference is PathwayLLM's reported zero-shot AUROC of 0.842, rising to 0.867 after light fine-tuning, both on retrospective data (PathwayLLM). We treat these as retrospective ceilings to compare against, not as numbers to beat under replay. GraphCare is included on mortality at $N=36$, its reported regime. It rarely appears in cross-site work, and we use it to test whether knowledge-graph grounding carries across sites at all. No reported result was found on Lacuna for GraphCare or MiME on eICU or HiRID.
Two ablations isolate the key ingredient. The first applies the availability restriction to every model, because it is model-agnostic. The second switches the grounding checker on and off with everything else fixed. The grounded summarizer counts as winning only if it beats tuned XGBoost on AUPRC and on PPV at the 2:1 alert budget under deployment-realistic data at $N \le 36$, with 20-draw intervals that do not overlap. It must also cut fabricated values toward zero from the roughly 6% rate PathwayLLM reports.
Why not just…?
Why not just use XGBoost with missingness flags and retrain locally? This is the real threat, and the proposal is designed so it can win. All models are local would predict exactly this outcome. The Record2Vec transfer evidence against numeric models was gathered on retrospective extracts, so it does not settle the question. If tuned trees match the text pipeline at $N=16$ under replay, we report that text portability was mostly retrospective completeness. That would be a useful negative result, and it would tell hospitals to deploy the cheaper model.
Why not just recalibrate the locked source model? Ötles et al. found that temporal shift mainly damages calibration while infrastructure shift damages ranking. A logistic recalibration of intercept and slope, fitted on $N$ local patients, is therefore the cheapest fix for part of the gap. We run it on top of every locked model. The decomposition shows directly how much of the availability term is ranking loss that recalibration cannot recover.
Why not skip the replay and go straight to silent mode? Silent mode at one partner site gives one site, one period and no control over $N$. It also cannot show which data fields drive the loss. The replay supplies many sites, controlled sample sizes and repeated draws. Silent mode is kept as the validity check for the replay, not replaced by it.
Decisive Checks
The first decisive check is the decomposition itself on eICU and HiRID, with MIMIC-IV as source. If availability shift is under a fifth of site shift for every method at every $N$, the confound hypothesis fails. The paper would then become a clean confirmation of the portability claims.
The second check is the learning-curve crossing: whether, and at what $N$, tuned trees overtake text representations under replay compared with under retrospective data. A shift in that crossing point is the practical finding.
The third check is fabrication. We count summary values absent from the input, stratified by how much data is missing. We predict that the ungrounded rate rises with missingness and that the grounded rate stays near zero without loss of AUPRC.
The fourth check is external validity of the replay. At a partner hospital, we run the best two configurations in silent mode for a fixed period. We then compare live-pipeline performance with replay-predicted performance for the same weeks, and with a later retrospective pull of those weeks, using the three-snapshot design of Ötles et al. If the replay predicts the live number better than the retrospective pull does, the emulation earns its place as a pre-deployment test. If it does not, we report which missingness patterns it failed to capture. This stage depends on securing a partner site and is scoped as conditional.
References
- Can we generate portable representations for clinical time series data using LLMs?
- GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs
- MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
- PathwayLLM: Explainable Clinical Trajectory Modeling with Structured Pathways for Sepsis Prediction
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- All models are local: time to replace external validation with recurrent local validation
- Preparing a Clinical Support Model for Silent Mode in General Internal Medicine
- Intersection of Artificial Intelligence and Medical Education (Student Abstract)
- Enhancing predictive modeling for respiratory support with LLM-driven guideline adherence
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning