Large Transformers pretrained on millions of patients' structured medical records adapt well from a handful of labeled examples. A Transformer is a sequence model in which every event can directly look at every other event through a learned weighting called attention. There is one stubborn exception. On EHRSHOT, a benchmark that gives a model only $k$ labeled patients per class, a 141M-parameter clinical Transformer beats gradient-boosted trees for $k \le 64$. On long-horizon tasks, such as predicting whether lupus or pancreatic cancer will be newly diagnosed within a year, the trees catch up or pull ahead once $k > 64$. Those trees run on simple code counts expanded through a medical ontology (EHRSHOT).
We think the cause is how the model is read out, not what it knows. The standard few-shot protocol trains a linear classifier on the model's final summary vector. A next-event objective trains that vector to predict what happens soon, not to tally evidence accumulated over years. We propose a small readout module that sits on the frozen Transformer. It has one attention "query" per node of a medical ontology (for example, one per diagnosis group), and each query may only attend to past events whose codes fall under that node. The module produces an ontology-organized, context-aware version of the counts that make trees strong, while keeping the pretrained representations that make the Transformer strong at small $k$.
Core Claim
A frozen clinical Transformer read out through ontology-restricted attention pooling will close the long-horizon diagnosis gap against count-based trees on EHRSHOT at $k \in \{64, 128\}$ without losing its advantage at small $k$. It needs no local pretraining and no backbone fine-tuning. The ontology mask is the ingredient we claim matters, and three ablations test it. The first is attention pooling without the mask. The second is masked mean pooling without learned attention. The third simply concatenates count features to the standard embedding.
We state openly that the third ablation could match the full method. If it does, that is itself a clean finding: the long-horizon gap is a readout-of-cumulative-evidence problem, and a two-line fix solves it. The experiments are designed so that either outcome is informative.
Where this sits in prior work
Several papers motivate the gap. EHRSHOT establishes the benchmark and the long-horizon crossover. It uses the ontology only to build features for the trees; the Transformer is read through a final-state linear probe. A multi-center study shows the same backbone gives a "13% improvement" over local trees with few labels. It needs continued pretraining on local unlabeled data to match fully trained trees with under 1% of labels. Our method targets hospitals that cannot run that continued pretraining. Sparse-autoencoder probing of an EHR Transformer finds that internal features are organized by how common a condition is, not by organ system. Rare conditions (<3% prevalence) get weak early-layer features. This is why we pool rare codes under shared ontology parents and read from several layers rather than only the last one.
Other work supplies ideas we reuse in a different place. MiME hard-codes the visit, diagnosis and treatment hierarchy into a recurrent model's embeddings and reports a "15% relative gain in PR-AUC" on its smallest data split. Its structure lives in a model trained from scratch; ours imposes structure only at readout on a frozen pretrained model. GraphCare builds patient knowledge graphs from UMLS and GPT-4 triples and runs a separate graph network, matching 2%-data baselines with 0.1% of the data. It also shows that noisy 2-hop expansions hurt. That constrains us to curated ontology parent-child edges at a fixed depth, with depth treated as a studied variable rather than a free expansion.
Two alternatives buy sample efficiency with large pretraining or external services, and neither suits a hospital without those resources. MOTOR buys it with time-to-event pretraining on tens of millions of records. Record2Vec buys it with text summaries from hosted LLMs, at a cost to privacy and fine timing.
Research Program
The readout works as follows. Let $h_i^{(\ell)}$ be the frozen Transformer's representation of event $i$ at layer $\ell$, and $c_i$ its code. For each ontology node $a$, $D(a)$ is the set of codes descending from $a$. The query $q_a$ is fixed: it is the mean of the pretrained input embeddings of codes in $D(a)$. It therefore needs no labels. The node summary is
$$z_a = \sum_{i \le t,\; c_i \in D(a)} \frac{\exp(q_a^\top W h_i^{(\ell)} + \beta\,\tau_i)}{\sum_{j \le t,\; c_j \in D(a)} \exp(q_a^\top W h_j^{(\ell)} + \beta\,\tau_j)}\, h_i^{(\ell)}.$$
Here $t$ is the prediction time, $\tau_i$ is log time elapsed before $t$, $\beta$ is one learned recency scalar, and $W$ is a shared low-rank matrix of rank 8 to 16.
The classifier sees, for each node, its summary $z_a$ plus the log count $\log(1+|\{i : c_i \in D(a)\}|)$. It also sees the standard final-state embedding. A group-lasso penalty over nodes lets the few-shot fit keep only a handful of ontology groups. The learned parameters are $W$, $\beta$ and the sparse linear weights, so the parameter count stays in the low thousands per task.
We will study four design choices on held-out development tasks only. They are ontology depth (1 versus 2 levels of SNOMED/ICD-style grouping), which layer is read, whether nodes are restricted to diagnoses or also include labs and medications, and how the group-lasso penalty is set when $k$ is tiny. We will then look at which ontology nodes the sparse classifier selects for each long-horizon task. This checks whether the model's evidence is clinically sensible.
Baselines
The naive reference is the released EHRSHOT protocol exactly as published. It consists of a logistic-regression probe on the frozen final-state embedding of CLMBR-T-base, and the count-based gradient-boosted trees with ontology expansion. For both we reuse the benchmark's fixed $k$-shot samples and replicate seeds. The trees are the strongest existing competitor on the long-horizon question, because they are the method that overtakes the Transformer there.
We found no reported per-task AUROC for either method on Lacuna. The summary gives only the qualitative crossover, with numbers in the source figures. We will therefore recompute both baselines ourselves on identical splits. Where the paper's released numbers differ from our recomputation, we will report both.
The strongest existing method under our resource constraint (no local pretraining) is off-the-shelf linear probing, which the multi-center study reports as a "13% improvement" over local trees with few labels. Continued pretraining from that study is reported as a reference ceiling that uses extra unlabeled data and compute; it is not a head-to-head competitor.
The ablations isolating the mask are:
- attention pooling over all events with the same $W$, count features and penalty, but no ontology restriction;
- masked mean pooling with $W = 0$;
- concatenation of the final-state embedding with the exact ontology-expanded count vector the trees use, fed to logistic regression and to the trees;
- the trees given our node summaries as extra features.
Every head gets the same hyperparameter search budget, 20 configurations chosen by cross-validation within the $k$-shot training set. There is no access to the test labels.
To count as beating the trees, the full method must match or exceed their mean AUROC on the long-horizon diagnosis tasks at $k = 64$ and $k = 128$. It must also stay within noise of the standard probe for $k \le 16$. To count as beating the concatenation ablation, it must win on long-horizon AUPRC by more than the replicate standard deviation.
Why not just…?
Why not just concatenate count features with the Transformer embedding? This is the real threat, and it is built in as the third ablation above. Counts capture accumulation but ignore context: two "chest pain" codes mean different things after a cardiac workup than before one. The masked attention can weight events inside a group by their pretrained representations. If concatenation matches us, we will report that the simpler fix suffices and that ontology-structured attention adds nothing at this scale.
Why not just fine-tune the whole Transformer, or add low-rank adapters? At $k \le 128$, updating millions of parameters invites the overfitting that small-cohort studies document. In the chest X-ray study, deeper networks overfit a 447-image dataset, and even shallow ones learned hospital shortcuts. We will nonetheless run low-rank adapter fine-tuning under the same tuning budget, so this is measured rather than assumed.
Why not just do continued pretraining on local data? It works: under 1% of labels suffices after local pretraining in the multi-center study. But it requires a large unlabeled local cohort and training compute, which is exactly the constraint we target. We treat it as a ceiling. If our readout recovers a meaningful share of that gain without any pretraining, that is the practical contribution.
Why not just use a time-to-event backbone like MOTOR? Its gains come from pretraining on tens of millions of records (MOTOR). The readout does not depend on the backbone, so where weights are available we will apply it on top of MOTOR too. This tests whether the benefit adds to a better pretraining objective or overlaps with it.
Proposed Experiments
The main experiment runs all 15 EHRSHOT tasks at all eight shot levels, reporting AUROC and AUPRC averaged over the benchmark's replicates. Results are broken out by task family: operational outcomes, lab anticipation, new diagnosis, and chest X-ray findings.
The decisive check is the long-horizon diagnosis family at $k \in \{64, 128\}$. We predict the gains concentrate there and in rare diagnoses, where pooling under ontology parents matters most. We predict little change on operational tasks, where the final-state probe already wins. A uniform small gain across all task families would weaken our explanation. A gain that appears only with the mask, and not in the unmasked or concatenation ablations, would support it.
The second check is cross-site transfer. We apply the Stanford-pretrained backbone with our readout to MIMIC-IV mapped to the common OMOP data schema, following the multi-center study. Few-shot mortality and readmission are evaluated without continued pretraining. Here the ontology mask may help because code groups transfer more stably across hospitals than individual local codes do.
The third check is a mechanistic one, following the sparse-autoencoder study. We stratify gains by the prevalence of the codes that the selected ontology nodes cover. This tests whether the readout helps specifically where the backbone's rare-condition features are weak.
References
- EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
- A multi-center study on the adaptability of a shared foundation model for electronic health records
- Probing Clinical Concepts in an EHR Foundation Model via Sparse Autoencoders
- MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
- GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs
- MOTOR: A Time-to-Event Foundation Model For Structured Medical Records
- Can we generate portable representations for clinical time series data using LLMs?
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data