Abstract
Identifying patients with rare diseases is a significant challenge due to extreme class imbalance and noisy administrative claims data. RareBERT is a specialized transformer-based model that introduces novel temporal and contextual embeddings alongside an adaptive loss function. It significantly outperforms existing medical models like Med-BERT, achieving nearly triple the precision (AUPRC) in identifying rare conditions and demonstrating remarkable robustness to data sparsity.
Introduction
Identifying patients with rare diseases is a massive challenge for global healthcare. Individually, these diseases affect approximately 1 in 1,500 people, but collectively they impact over 300 million people worldwide. Patients often endure years of misdiagnosis because symptoms are non-specific and clinical awareness is low. While machine learning has been applied to Electronic Health Records (EHR), administrative claims data presents unique hurdles: extreme class imbalance, unlabeled "noisy" data (where negatives may be undiagnosed positives), data sparsity, and a lack of specific ICD codes for many rare conditions.
Methodology
RareBERT builds upon the Med-BERT architecture but introduces three critical enhancements to handle the nuances of rare disease identification:
- Contextual Embeddings: Unlike standard models, RareBERT uses "Type Embeddings" to distinguish whether a code represents a diagnosis (Dx), a procedure (Px), or a treatment (Rx).
- Temporal Reference Embedding: It captures the exact position of an event in time relative to a specific "anchor date," allowing the model to understand the longitudinal progression of symptoms.
- Adaptive Loss Function: A novel loss function dynamically balances predicting "masked" medical events (pre-training) and classifying the disease (fine-tuning), improving convergence speed by roughly 66%.
- Positive Unlabeled (PU) Learning: To address undiagnosed patients in the "negative" set, the model integrates a PU-learning framework to iteratively isolate true negatives.
The study utilized a massive real-world dataset from Symphony Health’s Integrated Dataverse (IDV), focusing on X-Linked Hypophosphatemia (XLH) and Exocrine Pancreatic Insufficiency (EPI) as primary case studies.
Results and Discussion
RareBERT demonstrated a massive leap over existing benchmarks. On the XLH dataset, RareBERT achieved an 80.1% AUPRC, compared to just 30.0% for Med-BERT and ~43% for standard LSTM and XGB models. At a fixed precision of 10%, RareBERT identified 87.5% of rare disease patients, nearly doubling the identification rate of Med-BERT (46.4%).
One of the most striking findings was RareBERT’s robustness. When researchers simulated "noisy" data by randomly deleting 15% of medical events, RareBERT’s performance dropped by only 1.35%. In contrast, traditional models like Boosting Trees (XGB) saw their performance drop by over 67%.
Clinical validation through attention weights confirmed that RareBERT focuses on clinically relevant signals—such as the use of Vitamin D supplements and painkillers—which are known proxies for the target rare diseases.
Conclusion
While RareBERT is highly effective at spotting patterns in medical codes, it currently lacks access to demographic information and physician specialties. Future iterations should incorporate this metadata and explore more sophisticated imputation mechanisms to fill gaps in medical claims. Ultimately, the goal is to deploy RareBERT in clinical environments to help doctors flag potential rare disease cases years earlier than they do today.
Summary updated