TL;DR
This paper tackles the challenge of automatically understanding the content and urgency of electronic consults (eConsults) from primary care providers to specialists. By developing and evaluating BERT-based models, they demonstrate that domain-specific pre-trained models and multi-task learning significantly improve classification accuracy for both triage status and question type, offering a path towards more efficient healthcare resource allocation.
What Problem Does This Paper Address?
Electronic consult (eConsult) systems are increasingly used to streamline communication between primary care providers (PCPs) and specialists, aiming to reduce unnecessary in-person visits and improve access to care. However, these eConsult messages are unstructured text, making it difficult for healthcare systems to automatically understand their content and the urgency of the patient's needs. This lack of insight hinders efforts to improve specialist efficiency, guide resource allocation, and manage population health effectively. Specifically, the paper addresses two key challenges: 1) accurately classifying the triage status (urgent vs. non-urgent) assigned by specialist reviewers, and 2) identifying the underlying type of clinical question posed by the PCP. The latter is crucial for understanding the information needs driving these consults.
Methods
The authors leverage pre-trained transformer models, specifically BERT and its variants, for both classification tasks. For triage status classification, they initially train models on the entire dataset and then explore reviewer-specific models to investigate inter-reviewer consistency. For question type classification, they employ a multi-label classification approach due to the non-mutually exclusive nature of the question types.
Key methodological contributions include:
- Domain-Specific BERT Models: They compare standard BERT with domain-specific variants like SciBERT (pre-trained on scientific text) and Bio+Clinical BERT/Bio+Discharge Summary BERT (pre-trained on biomedical literature and clinical notes).
- Multi-Task Learning: For question type classification, they investigate a multi-task learning scheme where a single BERT model is fine-tuned to predict all question types simultaneously. This approach aims to leverage shared representations and improve generalization.
- Bag-of-N-grams (BoN) and Bag-of-CUIs (BoC) with SVMs: These traditional machine learning methods are used as baselines to compare against the performance of BERT models. BoC utilizes medical concept identifiers from the cTAKES system.
Datasets
The study utilizes de-identified text data from the San Francisco Department of Public Health (SFDPH) spanning from 2008-2017, focusing on gastroenterology and liver specialists. This dataset comprises over 33,000 eConsults.
For triage status classification, the automatic labels assigned by specialist reviewers are used.
For question type classification, a subset of 969 questions was manually annotated by a certified medical coder using an iterative process to define five question types: Diagnosis/Management (D/M), Procedure-Colonoscopy (P-C), Procedure-EGD (P-EGD), Procedure-Other (P-Other), and Procedure-FlexSig (P-F). The data was split into training, development, and test sets with a 70%/10%/20% ratio.
Key Insights and Ideas
- Triage Status is Difficult: Predicting triage status is challenging, and even more so when trying to generalize across different specialist reviewers. This suggests significant inter-reviewer variability in decision-making.
- Domain-Specific Models Shine: SciBERT, pre-trained on scientific text, demonstrates superior performance for question type classification compared to general BERT. This highlights the importance of domain-specific pre-training for capturing relevant medical terminology and nuances.
- Multi-Task Learning Benefits Vary: While multi-task learning shows promise, its benefit for question type classification is inconsistent, sometimes improving performance and sometimes decreasing it depending on the specific BERT variant.
- Synergy of Domain-Specific Vocabularies and Multi-Task Learning: The best results for question type classification are achieved by combining domain-specific BERT models (like SciBERT) with multi-task learning, suggesting a synergistic effect.
- Implicit Information Needs: The paper suggests that understanding question type classification can reveal implicit information needs of PCPs, which is valuable for improving specialist efficiency and education.
Key Results
- Triage Status Classification: Overall performance for triage status classification across all consults was modest (around 0.45-0.48 F1 score). Reviewer-specific models showed better performance when trained and tested on the same reviewer's data (diagonal of Table 2), but performance dropped significantly when tested on other reviewers, confirming inter-reviewer inconsistency.
- Question Type Classification:
- For separate classifiers (Table 3), SciBERT achieved the highest F1 scores across most categories, with a notable 0.92 for P-EGD.
- For multi-task learning (Table 4), SciBERT with multi-task learning achieved the best overall performance, outperforming other configurations. For instance, it achieved an F1 score of 0.89 for P-EGD and 0.41 for P-Other.
- The paper notes that P-O (Procedure-Other) is difficult to annotate reliably, reflected in its lower F1 scores.
Limitations and Future Work
- Inter-Reviewer Inconsistency: The paper acknowledges that the inconsistency in triage decisions among reviewers is a significant challenge. Future work could explore within-reviewer classifier performance to determine if the models themselves are the limitation or if reviewer behavior is inherently variable.
- Limited Data for Some Categories: The P-F (Procedure-FlexSig) category had only two instances in the evaluation split, highlighting the need for more annotations.
- Modeling Domain Structure: The authors suggest future work should explore more sophisticated modeling of domain structure, including understanding question text better and modeling relationships between output variables.
- Sequential Classification: A sequential classification approach could potentially incorporate logic for inferring implicit requests.
- Expanding Annotations and Specialties: The team plans to continue annotating more questions for question type, refine annotation guidelines, and expand to additional specialties.
- Investigating Classifier Choice: Future work should also investigate whether the chosen classifier models are the bottleneck for triage status prediction or if the inherent variability in reviewer decisions is the primary factor.
Summary updated