Abstract
Background: Recently, numerous foundation models pretrained on extensive data have demonstrated efficacy in disease prediction using Electronic Health Records (EHRs). However, there remains some unanswered questions on how to best utilize such models especially with very small fine-tuning cohorts. Methods: We utilized Med-BERT, an EHR-specific foundation model, and reformulated the disease binary prediction task into a token prediction task and a next visit mask token prediction task to align with Med-BERT's pretraining task format in order to improve the accuracy of pancreatic cancer (PaCa) prediction in both few-shot and fully supervised settings. Results: The reformulation of the task into a token prediction task, referred to as Med-BERT-Sum, demonstrates slightly superior performance in both few-shot scenarios and larger data samples. Furthermore, reformulating the prediction task as a Next Visit Mask Token Prediction task (Med-BERT-Mask) significantly outperforms the conventional Binary Classification (BC) prediction task (Med-BERT-BC) by 3% to 7% in few-shot scenarios with data sizes ranging from 10 to 500 samples. These findings highlight that aligning the downstream task with Med-BERT's pretraining objectives substantially enhances the model's predictive capabilities, thereby improving its effectiveness in predicting both rare and common diseases. Conclusion: Reformatting disease prediction tasks to align with the pretraining of foundation models enhances prediction accuracy, leading to earlier detection and timely intervention. This approach improves treatment effectiveness, survival rates, and overall patient outcomes for PaCa and potentially other cancers.
Introduction
Pancreatic cancer (PaCa) is a devastating disease often referred to as a "silent killer" due to its asymptomatic nature in early stages. Early detection is critical for improving survival rates, yet traditional screening methods are often insufficient. Deep learning models applied to Electronic Health Records (EHRs) offer a promising avenue for early risk assessment. However, state-of-the-art transformer-based foundation models, such as Med-BERT, typically require large amounts of annotated data for effective fine-tuning. This poses a significant challenge for rare diseases or specific cancers where high-quality labeled datasets are scarce.
This paper investigates how to maximize the predictive power of EHR foundation models in "few-shot" scenarios—where only a limited number of positive and negative patient examples are available. The core hypothesis is that aligning the downstream clinical prediction task with the model's original pretraining objective can unlock latent knowledge and improve performance without the need for massive fine-tuning datasets.
Methodology
The researchers compared three distinct architectural approaches for disease prediction using Med-BERT:
- Med-BERT-BC (Binary Classification): This is the conventional approach where a standard classification head is added to the model to output a binary probability (0 or 1) for the presence of cancer.
- Med-BERT-Sum (Token Prediction): This method leverages the pretrained embeddings of eight specific ICD-10 codes associated with pancreatic cancer. It calculates disease probability by performing a dot product between the patient's contextual representation and these specific disease tokens.
- Med-BERT-Mask (Next Visit Mask Token Prediction): This approach reformulates the prediction as a "fill-in-the-blank" task. The model treats the patient's future visit as a
[MASK]token and predicts the likelihood that one of the target PaCa ICD-10 codes will appear in that visit.
By using the [MASK] approach, the model performs a task nearly identical to its original pretraining (Masked Language Modeling), allowing it to utilize its existing understanding of medical code sequences more effectively.
Experimental Setup
The study utilized a pancreatic cancer cohort comprising 31,243 patients (12,273 cases and 18,970 controls) from the original Med-BERT v2 study. The data included structured EHR features such as diagnosis, medication, and procedure codes. The models were evaluated across various training sample sizes, ranging from extreme few-shot (10 samples) to the full dataset (~21,000 samples), using a 7:1:2 train/validation/test split.
Results and Discussion
The experimental results demonstrate the clear advantage of task alignment in low-data regimes:
- Few-Shot Performance: In scenarios with 10 to 500 training samples, Med-BERT-Mask significantly outperformed the standard Med-BERT-BC, with average Area Under the Curve (AUC) improvements of 3% to 7%.
- Data Efficiency: Med-BERT-Mask achieved an AUC of approximately 0.78 with only 200 total samples (100 cases, 100 controls). In contrast, the standard Med-BERT-BC required 1,000 samples to reach the same level of accuracy.
- Scaling Behavior: While Med-BERT-Mask excels in few-shot settings, its relative advantage diminishes as more data becomes available. For larger datasets (10,000+ samples), Med-BERT-Sum emerged as the top performer, peaking at an AUC of ~0.83.
These findings suggest that for rare diseases where data is limited, reformulating the task to match the pretraining objective is a superior strategy. However, as data becomes abundant, the model can benefit from more specialized classification heads that learn discriminative features specific to the target task.
Conclusion
This research highlights a powerful strategy for applying large-scale clinical foundation models to rare disease prediction. By reformatting disease prediction tasks to align with the pretraining objectives of models like Med-BERT, researchers can significantly enhance prediction accuracy and data efficiency. This approach paves the way for earlier detection and intervention in pancreatic cancer and potentially other conditions where training data is a bottleneck, ultimately leading to improved patient outcomes.
Future work should focus on validating these methods with raw clinical data, testing on more imbalanced real-world cohorts, and exploring the applicability of this technique across a broader range of diseases and foundation models.
Summary updated