TL;DR
This paper introduces SANSformers, a novel attention-free neural network architecture designed to improve forecasting of healthcare utilization from Electronic Health Records (EHRs). By combining specialized inductive biases with a self-supervised pre-training strategy (GSP), SANSformers achieve strong performance, particularly for smaller, divergent patient subgroups where traditional models struggle.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) are rich sources of information for predicting future healthcare needs. While powerful Transformer models have shown success in many domains, their application to EHRs is nuanced. The sequential and often sparse nature of EHR data, coupled with the computational cost of self-attention mechanisms, can make them less effective than simpler models, especially for smaller patient cohorts (e.g., those with rare diseases). These smaller groups often have unique health trajectories that are difficult to model with data-hungry, general-purpose architectures. The challenge is to develop models that are both accurate and efficient, capable of leveraging large datasets for general learning while also performing well on specialized, smaller subgroups.
Methods
The core innovation is the SANSformer architecture, which modifies the standard Transformer by replacing the computationally expensive self-attention mechanism with attention-free mixers. These mixers, inspired by recent work in computer vision and NLP, use simpler projection operations to capture token interactions. Key components include:
- Axial Decomposition: The input tensor is decomposed along its intra-visit and inter-visit dimensions, allowing for more efficient processing.
- Mixers (Visit Mixer and Causal Time Mixer): These replace self-attention to aggregate information within a visit and across visits, respectively. The Causal Time Mixer uses causal masking to prevent future information leakage.
- $\Delta\tau$ Embeddings: These explicitly encode the time difference between visits, capturing temporal dynamics more effectively than simple positional encodings.
- Generative Summary Pre-training (GSP): A novel self-supervised pre-training strategy. Instead of predicting the next token (which can be sensitive to noise in EHR data), GSP predicts summary statistics of future healthcare utilization (e.g., number of visits) based on past patient data. This allows the model to learn general patterns from a large population before being fine-tuned on specific subgroups.
Datasets
The authors used two real-world EHR datasets:
- Pummel: A confidential dataset from the Finnish Care Register for Health Care and Register of Primary Health Care visits, containing data for Finnish citizens aged 65 or older over seven years. This dataset was used for training and evaluating models on general populations and specific disease subgroups (Type 2 diabetes, Bipolar disorder, Multiple Sclerosis).
- MIMIC-IV v1.0: A publicly available dataset from Beth Israel Deaconess Medical Center, containing EHRs from approximately 250,000 patients. This dataset was used for evaluating the model on a mortality prediction task.
Key Insights and Ideas
- Attention-free models can be competitive for EHRs: The success of SANSformers demonstrates that the complex self-attention mechanism isn't always necessary for high performance in EHR tasks, offering a more efficient alternative.
- Inductive biases matter for EHR data: The specific architectural choices in SANSformers (axial decomposition, $\Delta\tau$ embeddings) are tailored to the unique characteristics of EHR data, leading to improved performance.
- Self-supervised pre-training is crucial for small subgroups: GSP significantly boosts performance for smaller, divergent patient cohorts, highlighting its value in overcoming data scarcity and unique trajectory challenges.
- Predicting summary statistics is robust to noise: GSP's approach of predicting summary statistics rather than raw sequences makes it more resilient to the inherent noise in EHR data during pre-training.
Key Results
- Superior performance: SANSformer models consistently outperformed or matched strong baselines (e.g., RETAIN, SARD) on various tasks, including predicting healthcare utilization (Pummel Visits and Diagnoses) and inpatient mortality (MIMIC-IV), even without pre-training.
- Data efficiency: In the Type 2 diabetes subgroup, SANSformers showed superior data efficiency, maintaining strong performance with smaller training datasets compared to other models.
- Subgroup performance boost: Pre-training with GSP provided a significant performance improvement for smaller, divergent subgroups like Bipolar disorder and Multiple Sclerosis, where randomly initialized models struggled. For instance, in the Multiple Sclerosis subgroup, Axial SANSformer + GSP achieved a Spearman correlation of 0.510 ± 0.017 for Pummel Visits.
- Scalability: SANSformers demonstrated good scalability, maintaining strong performance as model size increased, similar to traditional Transformers but with potentially better generalization.
Limitations and Future Work
- Sequence length limitation: SANSformers, like RNNs and Transformers, face practical limitations with very long sequences due to memory constraints and potential gradient issues.
- Proxy for cost: The model predicts healthcare utilization as a proxy for cost, rather than directly computing monetary costs, which might require further adaptation.
- Interpretability: The exclusion of self-attention might reduce interpretability compared to models that explicitly provide attention maps. The authors suggest exploring model-agnostic interpretability methods like SHAP.
- Future work: The authors suggest exploring feature interpretation from SANSformer models and comparing them with domain expert-engineered features. Further enhancements to directly predict healthcare costs are also proposed.
Summary updated