RareBERT: Transformer Architecture for Rare Disease Patient Identification using Administrative Claims

P. K. S. Prakash, Srinivas Chilukuri, Nikhil Ranade, Shankar ViswanathanAAAI 2021 · DOI

Abstract

Identifying patients with rare diseases is a significant challenge due to extreme class imbalance and noisy administrative claims data. RareBERT is a specialized transformer-based model that introduces novel temporal and contextual embeddings alongside an adaptive loss function. It significantly outperforms existing medical models like Med-BERT, achieving nearly triple the precision (AUPRC) in identifying rare conditions and demonstrating remarkable robustness to data sparsity.

Introduction

Identifying patients with rare diseases is a massive challenge for global healthcare. Individually, these diseases affect approximately 1 in 1,500 people, but collectively they impact over 300 million people worldwide. Patients often endure years of misdiagnosis because symptoms are non-specific and clinical awareness is low. While machine learning has been applied to Electronic Health Records (EHR), administrative claims data presents unique hurdles: extreme class imbalance, unlabeled "noisy" data (where negatives may be undiagnosed positives), data sparsity, and a lack of specific ICD codes for many rare conditions.

Methodology

RareBERT builds upon the Med-BERT architecture but introduces three critical enhancements to handle the nuances of rare disease identification:

  1. Contextual Embeddings: Unlike standard models, RareBERT uses "Type Embeddings" to distinguish whether a code represents a diagnosis (Dx), a procedure (Px), or a treatment (Rx).
  2. Temporal Reference Embedding: It captures the exact position of an event in time relative to a specific "anchor date," allowing the model to understand the longitudinal progression of symptoms.
  3. Adaptive Loss Function: A novel loss function dynamically balances predicting "masked" medical events (pre-training) and classifying the disease (fine-tuning), improving convergence speed by roughly 66%.
  4. Positive Unlabeled (PU) Learning: To address undiagnosed patients in the "negative" set, the model integrates a PU-learning framework to iteratively isolate true negatives.
RareBERT Embedding Architecture

The study utilized a massive real-world dataset from Symphony Health’s Integrated Dataverse (IDV), focusing on X-Linked Hypophosphatemia (XLH) and Exocrine Pancreatic Insufficiency (EPI) as primary case studies.

Results and Discussion

RareBERT demonstrated a massive leap over existing benchmarks. On the XLH dataset, RareBERT achieved an 80.1% AUPRC, compared to just 30.0% for Med-BERT and ~43% for standard LSTM and XGB models. At a fixed precision of 10%, RareBERT identified 87.5% of rare disease patients, nearly doubling the identification rate of Med-BERT (46.4%).

One of the most striking findings was RareBERT’s robustness. When researchers simulated "noisy" data by randomly deleting 15% of medical events, RareBERT’s performance dropped by only 1.35%. In contrast, traditional models like Boosting Trees (XGB) saw their performance drop by over 67%.

Robustness of RareBERT vs. Baselines

Clinical validation through attention weights confirmed that RareBERT focuses on clinically relevant signals—such as the use of Vitamin D supplements and painkillers—which are known proxies for the target rare diseases.

Conclusion

While RareBERT is highly effective at spotting patterns in medical codes, it currently lacks access to demographic information and physician specialties. Future iterations should incorporate this metadata and explore more sophisticated imputation mechanisms to fill gaps in medical claims. Ultimately, the goal is to deploy RareBERT in clinical environments to help doctors flag potential rare disease cases years earlier than they do today.

Summary updated

BibTeX
@inproceedings{prakash2021rarebert,
  author = {P. K. S. Prakash and Srinivas Chilukuri and Nikhil Ranade and Shankar Viswanathan},
  title = {{RareBERT: Transformer Architecture for Rare Disease Patient Identification using Administrative Claims}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16122},
}
Figure