TL;DR
This paper explores the practical challenges of building a real-world clinical NLP pipeline for extracting drug timelines from electronic health records. While state-of-the-art models perform well individually, combining them into a pipeline leads to unacceptable performance, even with domain adaptation techniques. The authors highlight model selection as a major barrier to improving overall pipeline performance.
What Problem Does This Paper Address?
The promise of modern NLP, particularly with large pre-trained models like BERT and RoBERTa, is to create reusable "building blocks" for complex information extraction tasks. This paper investigates the practical application of this idea in the clinical domain, specifically for extracting drug timelines from electronic health records (EHRs). The goal is to combine three NLP sub-tasks – negation detection, document time relation extraction (DocTimeRel), and aspectual link prediction (ALINK) – to understand drug start and stop events. The core challenge is that while individual models might perform well on their training data, their performance degrades significantly when applied to new, unseen clinical data (domain shift), and combining these imperfect models can lead to cascading errors, making the overall pipeline unusable.
Methods
The authors employ state-of-the-art neural methods, specifically fine-tuning large pre-trained transformer models (RoBERTa-base) for each of the three sub-tasks:
- Negation Detection: Identifying if a drug mention is negated (e.g., "not currently on statins"). They compare a feature-based Support Vector Machine (SVM) with a RoBERTa-based classifier.
- Document Time Relation Extraction (DocTimeRel): Determining the temporal relationship of a drug mention to the document's creation time (BEFORE, OVERLAP, AFTER, BEFORE/OVERLAP). Again, both SVM and RoBERTa-based models are used.
- Aspectual Link Extraction (ALINK): Classifying the temporal aspect of a drug mention (INITIATES, CONTINUES, TERMINATES, REINITIATES, NONE). A RoBERTa-based model is developed for this task, as no existing open-source models were found.
To address domain shift, they utilize unsupervised domain adaptation by continuing the language modeling pre-training objective on the target domain data before fine-tuning on the source data. This is referred to as "RoBERTa+LM."
Datasets
- THYME Colon Cancer Corpus: Used as the primary training dataset for all three NLP components.
- THYME Brain Cancer Corpus: Used as an out-of-domain dataset for evaluation.
- Internal Pulmonary Hypertension (PH) Notes: A newly annotated internal dataset used as a second out-of-domain evaluation set.
Key Insights and Ideas
- Pipeline Performance is Not the Sum of its Parts: Even when individual NLP components achieve good performance, their combination in a pipeline can lead to drastically reduced overall accuracy. This is a critical lesson for practical NLP system development.
- Domain Adaptation Helps, But Doesn't Solve Everything: While domain adaptation techniques (like continued pre-training on target data) show improvements on individual systems when moving to new domains, they don't fully bridge the performance gap.
- Model Selection is a Major Hurdle: The paper highlights that different models (SVM vs. RoBERTa) perform best on different tasks and datasets. In a real-world scenario without labeled target data for tuning, selecting the optimal model for each component becomes a significant, unsolved problem.
- Cascading Errors are a Real Threat: The paper implicitly demonstrates how errors in one stage of the pipeline can propagate and negatively impact subsequent stages, leading to an "unacceptable" final output.
Key Results
- In-Domain Performance: RoBERTa-based models generally outperformed SVMs on the THYME colon cancer data for all three tasks. Negation detection performance was particularly strong.
- Out-of-Domain Performance Degradation: Performance dropped significantly when testing on the THYME brain cancer and internal PH datasets.
- Domain Adaptation Impact: RoBERTa+LM showed gains on DocTimeRel for PH data and ALINK for both out-of-domain corpora, but the improvement was not always substantial and sometimes still lagged behind the SVM.
- Pipeline Accuracy ("All" column in Table 1): Mention-level accuracy for the combined pipeline was good on the in-domain data but dropped substantially for the out-of-domain datasets. For the internal PH data, the mention-level accuracy was as low as 0.506 with RoBERTa+LM, meaning roughly half of drug mentions had at least one attribute classified incorrectly.
Limitations and Future Work
- Model Selection: The authors explicitly state that model selection for domain adaptation when labeled target data is unavailable is an "unsolved and under-studied problem."
- Interpretability: The shift to "black box" pre-trained models makes it harder to understand the causes of domain shift. Future work should focus on interpretability research to better understand how different aspects of domains contribute to performance differences.
- Domain Variation: The paper suggests exploring how variations in institutions, specialties, note types, and authors contribute to domain shift and developing algorithms that can predict target domain performance without labeled data.
Summary updated