Abstract
The application of Transformer neural networks to Electronic Health Records (EHR) is challenging due to the distinct, multidimensional sequential structure of EHR data, often leading to underperformance when compared to simpler linear models. To overcome these challenges, we introduce SANSformer, a novel attention-free sequential model designed specifically with inductive biases to cater for the unique characteristics of EHR data. Our main application area is predicting future healthcare utilization, a crucial task for effectively allocating healthcare resources. This task becomes particularly difficult when dealing with divergent patient subgroups. These subgroups, characterized by unique health trajectories and often small in size, such as patients with rare diseases, require specialized modeling approaches. To address this, we adopt a self-supervised pretraining strategy, which we term Generative Summary Pretraining (GSP). GSP predicts summary statistics of a future window in the patient's history based on their past health records, thus demonstrating potential to deal with the noisy and complex nature of EHR data. We pretrain our models on a comprehensive health registry encompassing close to one million patients, before fine-tuning them for specific subgroup prediction tasks. In our evaluations, SANSformer consistently outshines strong EHR baselines. Importantly, our GSP pretraining method greatly enhances model performance, especially for smaller patient subgroups. Our findings underscore the substantial potential of bespoke attention-free models and self-supervised pretraining for enhancing healthcare utilization predictions across a broad range of patient groups.
Introduction
While Transformer models have revolutionized NLP and Computer Vision, their application to Electronic Health Records (EHR) has been inconsistent. EHR data consists of sequences of visits, each containing a "bag" of clinical codes, occurring at irregular intervals. This complexity often causes large neural networks to require massive amounts of data to outperform simple linear models, creating a significant gap when dealing with divergent patient subgroups—such as those with rare diseases—where data is scarce and health trajectories are unique.
Methodology
We introduce SANSformer (Sequential Analysis with Non-attentive Structures), which departs from the standard Transformer architecture to better suit medical data through three key innovations:
1. Attention-Free Mixers
Instead of computationally expensive self-attention, SANSformer uses "mixers" inspired by MLP-Mixer and FNet. These utilize simple linear projections and gating units to allow information flow across time and feature axes, providing a more stable alternative that is less prone to overfitting on small datasets.
2. Axial Decomposition and Temporal Encoding
The model processes data through two distinct branches: a Visit Mixer for "intra-visit" data (relationships between codes in a single visit) and a Causal Time Mixer for "inter-visit" data (evolution over multiple visits). Additionally, the model incorporates $\Delta\tau$ (delta-tau) embeddings to explicitly encode the time elapsed between consecutive visits.
3. Generative Summary Pre-training (GSP)
Standard token-level pre-training is often too noisy for EHR. GSP instead trains the model to predict future summary statistics (e.g., total visits in the next year) based on past history. This forces the model to learn high-level patient trajectories rather than individual clinical noise.
Experimental Results
The study utilized two major datasets: Pummel (1 million patients, 60 million visits) and MIMIC-IV (250,000 patients).
- Superior Accuracy: SANSformer consistently outperformed robust baselines like RETAIN, BEHRT, and SARD. On the Pummel visit prediction task, it achieved an $R^2$ of 0.311, compared to 0.298 for RETAIN and 0.272 for BEHRT.
- Massive Gains in Small Subgroups: For patients with Bipolar Disorder ($N=827$), pre-training with GSP boosted the Spearman correlation from 0.091 (random initialization) to 0.522.
- Scalability: Unlike RNNs or some Transformers that degrade as you add parameters, SANSformer maintained or improved performance as the number of parameters increased, even when training data size remained constant.
Discussion and Limitations
By removing attention, SANSformer loses traditional attention maps, though these are often considered unreliable for clinical interpretability. Future work should integrate model-agnostic tools like SHAP to explain decisions to clinicians. Additionally, while the model forecasts healthcare utilization as a proxy for cost, direct cost modeling remains a future objective. SANSformer demonstrates that for EHR, bespoke attention-free architectures combined with summary-based pre-training offer a robust path forward for clinical predictive modeling.
Summary updated