TL;DR
This paper explores how different Natural Language Processing (NLP) techniques can represent patient disease histories from electronic health records (EHRs) to predict future health outcomes. They found that sequence-based NLP models, particularly transformer architectures like EHR-BERT, outperform simpler methods in predicting outcomes like mortality and hospital admissions, even without task-specific fine-tuning.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) contain a wealth of longitudinal patient data, including sequences of disease codes. Effectively leveraging this data for predictive modeling is crucial for improving healthcare. However, representing these complex, time-ordered disease sequences in a way that captures meaningful clinical information for downstream tasks (like predicting mortality or future diagnoses) is challenging. Traditional epidemiological approaches often rely on manual grouping of disease codes, which is time-consuming and may lose granular information. This paper investigates whether advanced NLP methods, designed for text, can be adapted to learn rich, unsupervised representations of these disease sequences, and how these representations compare to simpler methods and to each other. A key question is whether using individual, granular disease codes is better than using pre-defined disease categories, and how different NLP architectures (bag-of-words vs. sequence-based) perform.
Methods
The researchers compared several NLP methods for generating patient representations (embeddings) from sequences of disease codes:
- Bag-of-Words Models:
- Latent Dirichlet Allocation (LDA): A topic modeling technique that assumes documents (patient sequences) are mixtures of topics.
- Doc2vec: An extension of word2vec that learns vector representations for sequences.
- Sequence-Based Transformer Models:
- Med-BERT: A BERT-based model pre-trained on ICD-9/10 codes.
- BEHRT: A BERT-based model pre-trained on disease categories, incorporating patient age.
- EHR-BERT (Novel Architecture): An extension of BEHRT that additionally incorporates sociodemographic information (gender, ethnicity, deprivation, calendar year) into the pre-training process.
They used two input strategies for these models:
- Disease Categories: A smaller set of 212 clinically grouped disease categories.
- Medcodes: A larger set of 9462 granular diagnostic codes.
The generated patient embeddings were then used as features in a simple logistic classifier to predict various clinical outcomes over a 1-year period. They also compared these embeddings against a baseline "Binary disease indicators" approach (where each disease is a binary feature) and a "Disease frequency counts" approach.
Datasets
- Clinical Practice Research Datalink (CPRD) Aurum: A large, nationally representative EHR dataset from UK general practices. The study included over 6.2 million adult patients registered on January 1, 2015, with at least two Long-Term Conditions (LTCs).
- Hospital Episode Statistics (HES) and Death Registrations: Used for linkage to secondary care data and mortality information.
The study used data recorded before January 1, 2015, for training the embeddings and data recorded in 2015 for evaluating predictive performance, ensuring the test data was unseen during training.
Key Insights and Ideas
- Sequence Matters: NLP models that consider the sequence of diseases (like transformers and doc2vec) generally outperform bag-of-words approaches (like LDA) in capturing predictive information. This highlights that the order in which conditions are diagnosed is clinically relevant.
- Transformers Lead the Pack: Transformer architectures (Med-BERT, BEHRT, and especially EHR-BERT) consistently showed the best predictive performance across various tasks, even when used without fine-tuning for specific outcomes. This suggests their ability to learn deep, contextual representations is highly effective.
- Sociodemographics Enhance Prediction: Incorporating sociodemographic information into the EHR-BERT model led to improved predictive performance, demonstrating the value of a holistic patient view beyond just clinical codes.
- Granularity vs. Categories: Surprisingly, using a smaller set of clinically grouped disease categories yielded similar predictive performance to using a much larger set of granular diagnostic codes. This implies that NLP models can effectively extract meaningful patterns from either level of detail, potentially reducing the burden of manual categorization.
- "Binary Disease Indicators" are a Strong Baseline: For certain tasks like predicting mortality, the simple "Binary disease indicators" approach performed comparably to some NLP models, suggesting that even basic representations can be effective, especially when interpretability is key.
Key Results
- Mortality Prediction: EHR-BERT (diseases) achieved the highest ROC-AUC of 0.866 and APS of 0.739 for predicting 60+ mortality.
- Healthcare Utilization: EHR-BERT also performed best for predicting ED attendance and emergency admissions, with ROC-AUCs of 0.646 and 0.739 respectively.
- New Disease Diagnosis: While transformer models generally outperformed others, the absolute predictive performance (APS) for new disease diagnoses was consistently low across all models, indicating a challenging prediction task.
- Disease Categories vs. Medcodes: For most models, the difference in performance between using disease categories (212) and Medcodes (9462) was minimal, suggesting that granular codes don't necessarily offer a significant predictive advantage when using advanced NLP.
- EHR-BERT's Advantage: EHR-BERT consistently outperformed BEHRT and Med-BERT, with absolute improvements in ROC-AUC ranging from 0.005 to 0.015 for the primary outcomes.
Limitations and Future Work
- Limited Fine-Tuning: The study focused on unsupervised representations without fine-tuning for specific prediction tasks. Fine-tuning could potentially further improve performance, as suggested by comparisons to prior work.
- Prediction Horizon: The study focused on 1-year prediction. Longer-term predictions might see different relative performance patterns.
- Data Imbalance: Event rates for some outcomes (e.g., new diagnoses) were low, which can impact model performance and evaluation metrics.
- Limited Input Data: The models primarily used coded disease sequences. Future work could explore incorporating other rich EHR data sources like symptoms, lab results, and prescriptions.
- Transformer Architectures: The authors suggest that transformer models, due to their ability to incorporate diverse information and reduce dimensionality, are promising for predictive applications and future research should explore their inclusion of additional EHR data inputs.
- Generalizability: While the study used a large dataset, further validation across different healthcare systems and coding ontologies would be beneficial.
- Bias and Fairness: The authors acknowledge the potential for bias when incorporating sociodemographic factors and emphasize the need for careful evaluation to assess and mitigate bias in clinical applications.
Summary updated