Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks

Ashvin Gupta, Brendan Delaney, Alessandra Russo

TL;DR

The tEHR framework demonstrates that "reading" electronic health records (EHR) as natural language is far more powerful than treating medical codes as arbitrary tokens. By converting structured clinical events into text streams and adapting pretrained Large Language Models (LLMs) like Mistral-7B, researchers achieved state-of-the-art results in predicting patient outcomes across both critical care and primary care settings.

What Problem Does This Paper Address?

Structured EHR data is a goldmine for predictive medicine, but it’s notoriously messy. Traditionally, models treat clinical concepts (diagnoses, labs, medications) as unique, dataset-specific tokens. For example, a model might see the code E11.9 and have to learn from scratch—purely through co-occurrence—that it relates to "Type 2 Diabetes."

This approach has two major flaws:

  1. Semantic Blindness: It ignores the rich medical knowledge already baked into the natural language descriptions of these codes.
  2. Heterogeneity: Different hospitals use different coding systems (ICD-10, SNOMED-CT, Read Codes). A model trained on one system often fails to generalize to another without intensive manual re-mapping.

The authors ask: Can we leverage the "prior knowledge" of a pretrained LLM to understand these records more effectively?

Methods

The authors introduce tEHR, a two-stage framework for adapting causal LLMs to longitudinal patient data:

  1. Clinical Narrative Textualization: Structured records are converted into a chronological text stream. Instead of a list of codes, the model "reads" sentences like: "Age 58 years; ... BMI 22.8 kg/m²; ... 2 weeks later Chronic kidney disease." (See Figure 1).
  2. Stage 1: Domain-Adaptive Continued Pretraining: The LLM (e.g., Mistral-7B) is fine-tuned on unlabelled patient trajectories using a standard next-token prediction loss. This helps the model adjust to the specific syntax and vocabulary of clinical narratives.
  3. Stage 2: Discriminative Fine-Tuning: For specific tasks (like mortality prediction), a special [PREDICT] token is appended to the patient history. A linear classification head is then trained on the hidden representation of this token using LoRA (Low-Rank Adaptation), keeping the backbone model mostly frozen.
Figure 1: Structured EHRs are converted into the MEDS event-based representation (Bert Arnrich et al.,

Datasets

The framework was evaluated on two distinct datasets to test its versatility:

  • MIMIC-IV: A public critical care dataset. Tasks included predicting ICU mortality, hospital mortality, and readmission.
  • CPRD (Clinical Practice Research Datalink): A private UK primary care dataset. The task was early detection of pancreatic cancer with lead times ranging from 1 to 12 months.

Both datasets were standardized using the MEDS (Medical Event Data Standard) format before textualization.

Key Insights and Ideas

  • Semantics are Essential: Through "lexical grounding" ablations, the authors found that replacing medical descriptions with meaningless "hashes" or scrambled text significantly degraded performance. This proves the model is actually utilizing its pretrained medical semantics, not just memorizing patterns.
  • Numbers Matter, Time is Implicit: Including numerical lab values was crucial for accuracy. However, explicitly writing out elapsed time (e.g., "5 days later") provided surprisingly little benefit. The model seems capable of inferring temporal relationships simply from the sequence of events.
  • Emergent Clinical Phenotypes: When the researchers visualized the model's internal patient representations using UMAP, they found that the model naturally clustered patients into clinically coherent groups—such as those with inflammatory, metabolic, or reproductive health patterns—without any explicit supervision for those categories.
UMAP visualization showing distinct clinical clusters in the learned representations.

Key Results

  • State-of-the-Art Performance: tEHR-Mistral-7B achieved the highest AUROC across all four MIMIC-IV tasks, notably reaching 0.887 for ICU mortality, outperforming specialized medical models like CLMBR-T and standard Transformer encoders.
  • Robust Cancer Prediction: On the CPRD dataset, tEHR outperformed all baselines in predicting pancreatic cancer, maintaining strong discriminative power even 12 months before diagnosis.
  • Scaling Wins: Larger backbones (7B-8B parameters) consistently outperformed smaller ones (0.6B), suggesting that the "medical common sense" of larger LLMs translates directly to better clinical predictions.
AUROC for pancreatic cancer prediction at different lead times.

Limitations and Future Work

  • Context Window Constraints: Textualizing records makes them much longer than code-based sequences. Even with a 16k token window, the authors had to truncate some long-term patient histories.
  • Inference Latency: While training costs were manageable, the inference speed of a 7B parameter LLM is significantly slower (approx. 300ms per patient) compared to lightweight models like XGBoost (0.2ms).
  • Future Directions: The authors suggest exploring cross-dataset transfer (e.g., training on MIMIC and testing on CPRD) to see if the natural language interface truly solves the "vocabulary mismatch" problem in healthcare AI.

Summary updated

Abstract

Structured Electronic Health Records (EHR) models typically treat each clinical concept as a dataset-specific token, discarding the natural-language semantics associated with the event. As a result, relationships between concepts must largely be learned from EHR co-occurrence, without directly exploiting any prior semantic knowledge. We investigate how pretrained causal language models can be adapted for longitudinal EHR prediction by developing tEHR. Here, we represent events with their natural language description and use this text stream to apply EHR-adaptive continued pretraining, followed by task-specific discriminative fine-tuning using LoRA, utilising the LLM as a longitudinal patient-encoder. We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance. Ablations show that performance declines when clinical descriptions are removed or misaligned, when numerical values are omitted, and when either stage of adaptation is removed, while explicit elapsed-time descriptions provide little additional benefit. Analysis of the learned patient representations further reveals clinically coherent structure associated with distinct patterns of disease risk and clinical history.
BibTeX
@misc{gupta2026evaluating,
  author = {Ashvin Gupta and Brendan Delaney and Alessandra Russo},
  title = {{Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks}},
  year = {2026},
  eprint = {2611.17053},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17053},
}
Figure