TL;DR
This paper introduces an open-source NLP framework designed to extract COVID-19 signs and symptoms from electronic health records (EHRs) across multiple institutions. It demonstrates the significant benefits of multi-site data for improving NLP model generalizability and highlights the challenges and necessity of federated approaches for annotation and evaluation in clinical research.
What Problem Does This Paper Address?
Clinical research often relies on extracting information from unstructured text within Electronic Health Records (EHRs). While Natural Language Processing (NLP) is a powerful tool for this, its adoption is hampered by inconsistencies in how data is documented across different healthcare institutions (ETL heterogeneity) and variations in how human annotators label data (human factor variations). These issues make it difficult to develop robust and generalizable NLP models, especially when working with data from multiple sites, which is crucial for broader research applicability. The paper specifically addresses the challenge of building and evaluating NLP models for clinical research using multi-site EHR data, focusing on COVID-19 signs and symptoms within the National COVID Cohort Collaborative (N3C).
Methods
The authors developed an open NLP framework that leverages the MedTagger NLP engine, a symbolic (rule-based) NLP processing tool. This approach was chosen for its transparency, interpretability, and deterministic outcomes, contrasting with many non-symbolic methods. For comparative purposes, they also implemented a deep learning-based NLP model using Bio+ClinicalBERT, fine-tuned for Named Entity Recognition (NER).
The framework supports:
- Symbolic NLP: Using SQL-based linkages to EHR data tables.
- Deep Learning NLP: Fine-tuning pre-trained models like Bio+ClinicalBERT.
- Federated Development: The core idea is to enable development and evaluation across multiple sites.
A key aspect of their methodology was the creation of a gold-standard dataset for COVID-19 signs and symptoms. This involved gathering clinical notes, de-identifying them, and then having a central annotation team create the labels. They also explored both single-site and multi-site model development and evaluation.
Datasets
The study utilized clinical notes from three institutions:
- Mayo Clinic
- University of Kentucky (UKen)
- University of Minnesota at Twin Cities (UMN)
These notes were filtered for positive COVID-19 test results and specific types of clinical encounters. A total of 369 de-identified clinical notes were randomly selected for annotation. The paper does not introduce a new benchmark dataset but rather uses a subset of data from the N3C initiative for their case demonstration.
Key Insights and Ideas
- Multi-site data is crucial for generalizability: NLP models trained or refined on data from multiple institutions show significantly improved performance and robustness compared to single-site models. This is because multi-site data captures a wider range of patient demographics and documentation practices.
- Symbolic NLP has advantages in federated settings: Due to their transparency and the fact that they don't embed PII, symbolic NLP models are easier to share and deploy in a federated manner, especially when data de-identification for statistical models is a significant hurdle.
- Federated annotation and evaluation are essential: The current centralized approach to annotation and evaluation presents significant challenges, including variations in inter-annotator agreement across sites and logistical difficulties. A federated approach, where annotation and evaluation happen locally at each site, is needed to better incorporate site-specific factors and improve consistency.
- Deep learning models struggle with small multi-site datasets: The performance of the deep learning model was limited by the relatively small size of the multi-site training data, highlighting the current advantage of symbolic methods in such scenarios.
Key Results
- Improved performance with multi-site data: The multi-site NLP algorithm generally outperformed single-site algorithms. For the rule-based model, multi-site training led to improved performance across all sites compared to single-site training on that specific site's data.
- Symbolic NLP outperformed deep learning: In this specific case study, the symbolic NLP models significantly outperformed the deep learning models for concept extraction, primarily attributed to the limited size of the multi-site labeled dataset for the deep learning model.
- Inter-Annotator Agreement (IAA) variations: The F1-score IAA for the annotated corpus showed considerable variation across sites: 0.686 for Mayo, 0.521 for UMN, and 0.373 for UKen, underscoring the challenges of consistent annotation across different institutions.
Limitations and Future Work
Limitations: * Centralized Annotation: The annotation process was centralized at Mayo Clinic due to resource constraints, leading to variations in IAA and potential biases. * Data Acquisition Challenges: Obtaining de-identified data from multiple sites for training deep learning models was difficult and time-consuming. * Small Multi-site Datasets: The sample sizes for fine-tuning deep learning models and for multi-site annotation were limited, impacting their performance. * De-identification Bottleneck: The process of de-identifying data for statistical models remains a significant bottleneck for multi-site research.
Future Work: The authors plan to prioritize the development of solutions for:
- Federated Annotation and Evaluation: Implementing mechanisms for decentralized annotation and evaluation to better handle site-specific factors.
- Systematic NLP Error Analysis: Developing a standardized framework and common definitions for analyzing NLP errors across different sites.
- Further Development of OHNLP Toolkit: Enhancing their toolkit to address these identified challenges.
Summary updated