TL;DR
This paper introduces a novel machine reading approach using finetuned language models to automatically extract crucial seizure frequency and seizure freedom information from unstructured epilepsy clinic notes. This method achieves near-human performance, paving the way for more efficient clinical decision-making and large-scale retrospective research in epilepsy.
What Problem Does This Paper Address?
The electronic health record (EHR) is a rich source of patient information, but extracting specific clinical details from unstructured notes is a significant challenge. For epilepsy patients, two of the most critical outcome measures are seizure freedom (whether they are experiencing seizures) and seizure frequency (how often they occur). Manually extracting this information from clinical notes is time-consuming and impractical for large-scale research or real-time clinical decision support. Existing NLP methods have shown promise, but accurately and efficiently extracting these specific, dynamic clinical outcomes from free-text notes remains an open problem.
Methods
The researchers developed a finetuning pipeline for pretrained neural language models. This pipeline involves several key steps:
- Unsupervised Domain Adaptation: Pretrained models (BERT, Bio_ClinicalBERT, and RoBERTa) were first exposed to a large corpus of 78,000 unannotated epilepsy clinic notes. This step, using masked language modeling (MLM), helps the models learn the nuances of clinical language specific to epilepsy.
- Task-Specific Finetuning: The models were then finetuned on publicly available datasets designed for question-answering tasks: BoolQ3L for classification (seizure freedom) and SQuADv2 for text extraction (seizure frequency and date of last seizure).
- Supervised Finetuning: Finally, the models were finetuned on a smaller, human-annotated dataset of epilepsy notes, specifically tailored to the tasks of interest.
This multi-stage finetuning process, inspired by previous work, aims to leverage the general language understanding of pretrained models and adapt them to the specific clinical context and tasks.
Datasets
- Unannotated Clinical Notes: Approximately 79,000 progress notes from epilepsy patients at the University of Pennsylvania Health System (UPHS) between January 1, 2015, and December 31, 2018, were used for unsupervised domain adaptation.
- Publicly Available Datasets: BoolQ3L (for classification) and SQuADv2 (for text extraction) were used for task-specific finetuning.
- Human-Annotated Dataset: A newly created dataset of 1,000 epilepsy clinic note paragraphs was manually annotated by 15 annotators (clinicians and non-clinicians) to serve as the ground truth for training and testing. This dataset was split into a training set of 700 paragraphs and a testing set of 300 paragraphs.
Key Insights and Ideas
- Near-Human Performance is Achievable: The finetuned models demonstrated performance comparable to human annotators for extracting critical epilepsy outcome measures, which is a significant step towards automating this process.
- Domain-Specific Pretraining Matters: Bio_ClinicalBERT, which is pretrained on clinical and biomedical texts, showed strong performance, highlighting the benefit of domain-specific pretraining for clinical NLP tasks.
- Minimal Annotation is Sufficient: A surprising finding was that a substantial portion of the performance gains from finetuning could be achieved with a relatively small amount of annotated data (around 70 notes). This makes the approach highly practical for new clinical questions.
- MLM and Annotations are Key: Ablation studies revealed that both the unsupervised domain adaptation (MLM) and the supervised finetuning on annotated data were crucial for achieving optimal performance. The annotated data, in particular, was found to be the most significant factor.
- Challenges in Temporal and Logical Reasoning: The models, while powerful, still struggled with complex temporal reasoning and understanding nuanced semantic implications, which are common challenges in NLP for clinical text.
Key Results
- Seizure Freedom Classification: Finetuned models achieved near-human accuracy, with BERTFT and Bio_ClinicalBERTFT reaching over 80% accuracy.
- Seizure Frequency and Date Extraction: All three finetuned models achieved human-level performance, with overall F1 scores exceeding 0.80 for extracting seizure frequency and the date of the last seizure.
- Best Model Combination: Bio_ClinicalBERTFT was identified as the best model for classification, while ROBERTAFT excelled at text extraction.
- Data Efficiency: Approximately 70 annotated notes were sufficient to achieve a significant portion (over 80%) of the final model performance for both classification and text extraction tasks.
Limitations and Future Work
- Generalizability: The study was conducted at a single healthcare center using specific neurology notes. The generalizability of the models to other institutions or departments needs to be tested and may require additional finetuning.
- Model Size Constraints: The researchers were limited by GPU memory, preventing them from experimenting with larger, potentially more powerful models like Longformer.
- Definition of Seizure Freedom: The definition of "recent seizures" (since last office visit or within 1 year) was pragmatic but arbitrary from a machine learning perspective and could influence classification accuracy.
- Seizure Type Granularity: The current models lump together all seizure types. Future work could aim to extract frequency information for different, clinically distinct seizure types.
- Inconsistent Data Entry: The presence of copy-pasted text and internally contradictory information in EHRs remains a challenge for NLP models.
Future work includes deploying this model for real-time clinical decision-making, conducting large-scale retrospective epilepsy research, and adapting the pipeline to answer new clinical questions with minimal data.
Summary updated