Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks
Lists
Only you see this
Abstract
Structured Electronic Health Record (EHR) models typically represent clinical concepts as dataset-specific tokens, discarding their associated natural-language semantics. Consequently, relationships between concepts must largely be learned from EHR co-occurrence without directly exploiting pretrained linguistic knowledge. We investigate the contribution of pretrained language semantics to longitudinal EHR prediction through tEHR, a framework that converts clinical events into natural-language descriptions and adapts pretrained autoregressive language models through EHR-specific continued pretraining and task-specific discriminative fine-tuning. We evaluate tEHR across four MIMIC-IV prediction tasks, where Mistral-7B achieves the strongest overall performance against structured-data, code-based, and LLM-based baselines. Controlled ablations demonstrate the importance of lexical grounding, numerical values, and both adaptation stages. A capacity-controlled experiment further shows that replacing clinical descriptions with atomic code tokens substantially reduces performance despite retaining the same LLM backbone. Alternative approaches incorporating pretrained semantic embeddings offer lower computational costs but do not match full textualisation. These results highlight the importance of both semantic representation and downstream adaptation when applying pretrained language models to structured EHR prediction.
BibTeX
@misc{gupta2026evaluating,
author = {Ashvin Gupta and Brendan Delaney and Alessandra Russo},
title = {{Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks}},
year = {2026},
eprint = {2611.17053},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.17053},
}