NLP Methods for Extraction of Symptoms from Unstructured Data for Use in Prognostic COVID-19 Analytic Models

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Greg M. Silverman, Himanshu S. Sahoo, Nicholas Ingraham, Monica Lupei, Michael A. Puskarich, Michael Usher, James Dries, Raymond L. Finzel, Eric Murray, John Sartori, György J. Simon, Rui Zhang, Genevieve B. Melton, Christopher J. Tignanelli, Serguei V. S. PakhomovJAIR 2021 · DOI

TL;DR

This paper introduces and evaluates two Natural Language Processing (NLP) methods for automatically extracting COVID-19 symptoms from unstructured clinical notes. These methods, a UMLS-based approach and a hybrid gazetteer-based approach, demonstrate the potential of symptom data to build prognostic models, offering a valuable tool for clinical decision support and telemedicine, especially in resource-constrained environments.

What Problem Does This Paper Address?

During the COVID-19 pandemic, accurately assessing patient prognosis and allocating resources became critical. However, crucial patient symptom information is often buried in unstructured clinical notes, making it difficult to access for real-time analysis and decision-making. This study aims to bridge this gap by developing and comparing NLP techniques to efficiently extract these symptoms, enabling the creation of prognostic models that can inform clinical care and support remote patient management.

Methods

The authors explored two primary NLP methods for symptom extraction:

  • UMLS-based Concept Extraction: This method utilizes existing "out-of-the-box" clinical NLP annotation systems (BIOMEDICUS, CLAMP, CTAKES, METAMAP) that map extracted terms to concepts within the Unified Medical Language System (UMLS). An ensemble of these systems was created using a Boolean combination to improve overall performance.
  • Hybrid COVID Symptoms Gazetteer: This approach combines a rule-based gazetteer (a custom lexicon of 164 COVID-19 symptoms) with UMLS mapping. It leverages spaCy's PhraseMatcher and EntityLinker for efficient phrase detection and UMLS disambiguation, along with a negation detection module.

Both methods were evaluated against a manually annotated reference set and compared for extraction performance and run-time efficiency.

Datasets

The study utilized a dataset of 19,924 Emergency Department (ED) visits from 10,110 unique patients confirmed as COVID-19 positive at M Health Fairview between March 1, 2020, and December 17, 2020. A subset of 5,006 patients older than 18 years was used for the analysis. A manually annotated reference set of 46 ED notes was created for baseline performance evaluation.

Key Insights and Ideas

  • Symptomatology as a Powerful Predictor: The study strongly suggests that extracted symptom data, even when used alone, can be as effective as vital signs in building prognostic models for outcomes like in-hospital mortality. This highlights the significant predictive power of patient-reported symptoms.
  • Hybrid Gazetteer's Efficiency: The custom "HYBRID COVID SYMPTOMS GAZETTEER" demonstrated superior run-time performance compared to the UIMA-based systems, making it a promising candidate for high-volume, near real-time symptom extraction.
  • Language Barriers Impact Prognosis: A significant finding was the disparity in outcomes between English and non-English speaking patients, particularly concerning fatigue, which showed opposing risk signals for mortality. This underscores the importance of considering language in prognostic modeling and highlights potential biases in data collection.
  • Telemedicine Potential: The ability to extract prognostic information from symptoms alone supports the expansion of telemedicine for COVID-19 triage and care, reducing the need for in-person visits and conserving resources.

Key Results

  • Extraction Performance: The Boolean ensemble achieved an overall weighted micro-average F1-score of 0.87. The HYBRID COVID SYMPTOMS GAZETTEER also performed well with an F1-score of 0.85, comparable to the other UIMA-based systems.
  • Run-Time Efficiency: The HYBRID COVID SYMPTOMS GAZETTEER was significantly faster than the UIMA-based systems, processing 3,000 ED notes in 1,913.80 seconds, compared to BIOMEDICUS (2,445.33 seconds) and METAMAP (81,328.53 seconds).
  • Prognostic Value: Models built using symptoms alone showed comparable performance to models using vital signs for in-hospital mortality. Symptoms were associated with various outcomes, such as dyspnea increasing the risk of in-hospital mortality (OR 1.85) and ventilation (OR 2.50).
  • Language Disparities: Non-English speakers showed a higher risk of mortality associated with fatigue (OR 1.95) compared to English speakers (OR 0.63).

Limitations and Future Work

  • Generalizability: The study's data was collected from a single healthcare network and primarily included white patients, limiting generalizability. The focus was also limited to CDC-defined symptoms, potentially missing others.
  • Manual Annotation: The manually annotated reference set was small (46 notes), which may not be representative of the entire patient population.
  • NLP Methods: The study did not explore all possible Boolean combinations for ensembling. The HYBRID COVID SYMPTOMS GAZETTEER's initial implementation had issues with negation detection.
  • Future Work: The authors plan to expand the lexicon for COVID-19 and Post-Acute Sequelae of COVID-19 (PASC), explore more sophisticated NLP methods (like BERT), improve negation handling, and create larger, more comprehensive manually annotated reference sets. They also aim to investigate the impact of language barriers further and explore the use of symptomatology for clinical decision support in various scenarios.

Summary updated

BibTeX
@article{fiahsi2021nlp,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Greg M. Silverman and Himanshu S. Sahoo and Nicholas Ingraham and Monica Lupei and Michael A. Puskarich and Michael Usher and James Dries and Raymond L. Finzel and Eric Murray and John Sartori and György J. Simon and Rui Zhang and Genevieve B. Melton and Christopher J. Tignanelli and Serguei V. S. Pakhomov},
  title = {{NLP Methods for Extraction of Symptoms from Unstructured Data for Use in Prognostic COVID-19 Analytic Models}},
  journal = {JAIR},
  year = {2021},
  doi = {10.1613/jair.1.12631},
}
Figure