Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models

Yogesh Kumar, Alexander Ilin, Henri Salo, Sangita Kulathinal, Maarit K. Leinonen, Pekka MarttinenIEEE Trans. Artif. Intell 2024 · DOI

TL;DR

This paper introduces SANSformers, a novel, attention-free neural network architecture designed to predict future healthcare utilization from electronic health records (EHRs). By employing self-supervised pre-training (GSP) and specialized inductive biases, SANSformers achieve superior performance, especially for smaller, divergent patient subgroups, outperforming traditional Transformer models.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are rich, sequential data sources crucial for predicting healthcare utilization. While Transformer models have shown great success in other domains, their application to EHRs is nuanced. The unique, often sparse, and temporally distant nature of EHR data can make complex models like Transformers less effective than simpler, feature-engineered models, especially for smaller patient subgroups (e.g., those with rare diseases). These subgroups present a significant challenge because training large models on limited data can lead to poor generalization and inaccurate predictions. Existing methods often struggle to balance performance across diverse patient populations, leading to potential inequities in healthcare resource allocation.

Methods

The core innovation is the SANSformer architecture, which adapts the Transformer's success while removing the computationally expensive self-attention mechanism. Instead, it uses attention-free mixers for processing sequential data. Key components include:

  • Axial Decomposition: The input tensor is decomposed along its temporal and intra-visit dimensions, allowing for more efficient processing.
  • Mixers (Visit Mixer & Causal Time Mixer): These replace self-attention, using channel and spatial projections to capture interactions within visits and across time steps, respectively.
  • $\Delta\tau$ Embeddings: These explicitly encode the time elapsed between visits, capturing crucial temporal dynamics.
  • Generative Summary Pre-training (GSP): A novel self-supervised pre-training strategy. Instead of predicting the next token (which can be sensitive to noise in EHR data), GSP predicts summary statistics of future healthcare utilization (e.g., number of visits) based on past data. This allows the model to learn robust representations from a large general population before fine-tuning on specific, smaller subgroups.

Datasets

  • Pummel Dataset: A confidential dataset from the Finnish Care Register for Health Care and Register of Primary Health Care visits, spanning seven years (2012-2018) and including pseudonymized EHRs of Finnish citizens aged 65 or older. This dataset was used for training and evaluating models on general populations and specific disease subgroups (Type 2 diabetes, Bipolar disorder, Multiple Sclerosis).
  • MIMIC-IV Dataset: A publicly available dataset from Beth Israel Deaconess Medical Center, comprising EHRs from approximately 250,000 patients. This dataset was used for evaluating the model's performance on hospital admissions, diagnoses, procedures, and mortality prediction.

Key Insights and Ideas

  • Attention-Free is Viable for EHR: The study demonstrates that removing self-attention and replacing it with mixers can maintain or even improve performance in EHR tasks, while significantly reducing computational complexity.
  • Self-Supervised Pre-training is Crucial for Subgroups: GSP pre-training on a large general population dataset provides a substantial performance boost for predicting healthcare utilization in smaller, divergent patient subgroups, outperforming models trained from scratch or with other pre-training methods.
  • Inductive Biases Matter: The specific architectural choices in SANSformers, such as axial decomposition and $\Delta\tau$ embeddings, are tailored to the unique characteristics of EHR data and contribute significantly to its effectiveness.
  • Scalability Without Overfitting: SANSformers exhibit strong scalability, improving performance with increased model size without the tendency to overfit that is observed in some baseline models, even with fixed training data.

Key Results

  • Superior Performance: SANSformers consistently outperformed established baselines (Lasso, RETAIN, SARD) on various EHR tasks, including predicting healthcare utilization (Pummel Visits and Diagnoses) and inpatient mortality (MIMIC-IV), especially when using GSP pre-training.
  • Data Efficiency: In the Type 2 diabetes subgroup, SANSformers demonstrated superior data efficiency, achieving strong performance with fewer training samples compared to other models.
  • Subgroup Performance: For smaller, challenging subgroups like Multiple Sclerosis, GSP-pretrained SANSformers matched or surpassed other models, highlighting their robustness and adaptability. For instance, on the MS subgroup, Axial SANSformer + GSP achieved a Spearman correlation of 0.510 ± 0.017 for Pummel Visits, outperforming other models.
  • Scalability: Both additive and axial SANSformer variants showed strong performance scalability on both Pummel and MIMIC datasets, maintaining effectiveness as model size increased.

Limitations and Future Work

  • Sequence Length Restriction: SANSformers, like RNNs and Transformers, face limitations in processing extremely long sequences due to memory constraints and potential vanishing gradient issues.
  • Proxy for Cost: The model predicts healthcare utilization (e.g., visit counts) as a proxy for cost, rather than directly predicting monetary costs, which may require further adaptation.
  • Interpretability: The removal of self-attention, while beneficial for efficiency, reduces the direct interpretability offered by attention maps. The authors suggest exploring model-agnostic interpretability methods like SHAP for future work.
  • Future Directions: The authors suggest exploring feature interpretation from SANSformer models and comparing them with expert-engineered features. Further enhancements to directly predict healthcare costs are also proposed.

Summary updated

BibTeX
@article{kumar2024self,
  author = {Yogesh Kumar and Alexander Ilin and Henri Salo and Sangita Kulathinal and Maarit K. Leinonen and Pekka Marttinen},
  title = {{Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2024},
  doi = {10.1109/tai.2024.3353164},
}
Figure