Assessing data availability and quality within an electronic health record system through external validation against an external clinical data source

Ellen L. Palmer, John H. Higgins, Saeed Hassanpour, James Sargent, Christina M. Robinson, Jennifer A. Doherty, Tracy OnegaBMC Medical Informatics Decis. Mak 2019 · OpenReview

TL;DR

This paper presents an informatics pipeline that extracts smoking status, pack year history, and cessation dates from unstructured clinical notes in Electronic Health Records (EHRs). By validating this pipeline against an external questionnaire, the study demonstrates its potential to improve patient identification for smoking cessation and lung cancer screening, despite significant data gaps in current EHRs.

What Problem Does This Paper Address?

Smoking remains a leading cause of preventable illness and death, yet its documentation in EHRs often lacks the detail needed for effective clinical action. While algorithms exist to extract smoking behaviors from clinical notes, their real-world performance and reliability when compared to external, trusted data sources have been poorly understood. This gap hinders the development of robust "smokers' registries" that could proactively identify patients for crucial interventions like smoking cessation counseling and lung cancer screening. The paper aims to bridge this gap by assessing the quality and availability of smoking-related data extracted from EHRs against an external clinical data source.

Methods

The researchers developed and evaluated an informatics pipeline designed to extract three key pieces of smoking-related information from clinical notes:

  • Smoking Status: Using a multi-class support vector machine algorithm adapted from previous work, the pipeline classifies notes into categories like "current," "former," "never," and "unknown." This involved identifying "hotspot" words related to smoking and analyzing surrounding text.
  • Pack Year History: A regular expression algorithm was employed to parse sentences containing smoking-related terms and extract numerical values representing frequency (packs per day) and duration (years smoked). These were then multiplied to calculate pack years.
  • Cessation Date: Sentences containing keywords like "quit" or "stop" were extracted, and any associated dates were converted to a uniform format.

The pipeline was trained and validated on annotated clinical notes. For external validation, the pipeline's output was compared against data from the New Hampshire Colonoscopy Registry (NHCR) questionnaire. The study also explored combining the NLP-extracted data with existing semi-structured EHR fields to assess if this improved performance.

Datasets

  • Training and Validation Data: 756 annotated clinical notes from a local Epic EHR data warehouse were used. This included 533 notes for training and 223 for validation.
  • External Validation Data: Data from 3,000 participants of the New Hampshire Colonoscopy Registry (NHCR) was used. This registry collects comprehensive colonoscopy data and includes patient questionnaires with smoking status, pack year history, and cessation information. For this study, 1,504 NHCR participants were successfully matched to clinical notes.

Key Insights and Ideas

  • Significant Data Gaps in EHRs: A striking finding is that a substantial majority (73%) of the analyzed clinical notes contained no smoking behavior information, highlighting a major challenge for leveraging EHR data for smoking-related interventions.
  • NLP as a Complement, Not a Replacement: While NLP can extract valuable information from unstructured notes, it's most effective when combined with existing semi-structured EHR data. This "fact-checking" approach improves reliability.
  • External Validation is Crucial: Assessing NLP algorithms against an external, independent data source (like the NHCR questionnaire) is essential for understanding their true clinical utility and performance in real-world scenarios, beyond just comparing against human annotations.
  • Lung Cancer Screening as a Viable Use Case: The pipeline demonstrated promise in identifying patients eligible for lung cancer screening, a critical public health initiative. Even with limitations, the ability to flag potential candidates is a significant step.

Key Results

  • Data Availability: 73% of the 1,504 matched clinical notes contained no smoking behavior information.
  • Smoking Status Concordance: The weighted Cohen's kappa between the external questionnaire and the pipeline-identified smoking status was 0.56 (for pipeline-only data) and 0.62 (for combined pipeline and semi-structured data).
  • Pack Year Correlation: The correlation between pack year history reported by the combined EHR sources and the external questionnaire was 0.39 (n=81).
  • Lung Cancer Screening Eligibility:
    • Combined EHR data (pipeline + semi-structured): Positive Predictive Value (PPV) of 85.4%, Negative Predictive Value (NPV) of 83.8%, Sensitivity of 63.1%, and Specificity of 94.7%.
  • Pipeline-only data: Showed a significant drop in sensitivity (20.9%) compared to the combined approach.

Limitations and Future Work

  • Data Missingness: The most significant limitation is the high rate of missing smoking behavior information in EHRs.
  • Pack Year Underestimation: The pipeline sometimes underestimated pack year history due to misclassifying frequency or duration as full pack years. The authors suggest heuristic cutoffs or further algorithm refinement.
  • Cessation Date Limitations: The external questionnaire did not capture cessation dates, preventing a full validation of that pipeline component.
  • Cross-sectional Design: The study used a cross-sectional design; future work should explore longitudinal data.
  • Generalizability: The use of a cancer screening questionnaire for external validation might introduce bias, and the patient population may not be fully representative of all clinical settings.
  • Future Directions: The authors propose external validation on different hospital systems, whole-record classification to reduce missing data, and developing the software for wider integration into EHRs and for public release. They also emphasize the need to identify more use cases and develop comprehensive reporting for clinicians.

Summary updated

Abstract

Background Approximately 20% of deaths in the US each year are attributable to smoking, yet current practices in the recording of this health risk in electronic health records (EHRs) have not led to discernable changes in health outcomes. Several groups have developed algorithms for extracting smoking behaviors from clinical notes, but none of these approaches were assessed with external data to report on anticipated clinical performance. Methods Previously, we developed an informatics pipeline that extracts smoking status, pack year history, and cessation date from clinical notes. Here we report on the clinical implementation performance of our pipeline using 1,504 clinical notes matched to an external questionnaire. Results We found that 73% of available notes contained no smoking behavior information. The weighted Cohen’s kappa between the external questionnaire and EHR smoking status was 0.62 (95% CI 0.56–0.69) for the clinical notes we were able to extract information from. The correlation between pack years reported by our pipeline and the external questionnaire was 0.39 on the 81 notes for which this information was present in both. We also assessed for lung cancer screening eligibility using notes from individuals identified as never smokers or smokers with pack year history extracted by our pipeline (n = 196). We found a positive predictive value of 85.4%, a negative predictive value of 83.8%, sensitivity of 63.1%, and specificity of 94.7%. Conclusions We have demonstrated that our pipeline can extract smoking behaviors from unannotated EHR notes when the information is present. This information is reliable enough to identify patients most likely to be eligible for smoking related services. Ensuring capture of smoking information during clinical encounters should continue to be a high priority.
BibTeX
@inproceedings{palmer2019assessing,
  author = {Ellen L. Palmer and John H. Higgins and Saeed Hassanpour and James Sargent and Christina M. Robinson and Jennifer A. Doherty and Tracy Onega},
  title = {{Assessing data availability and quality within an electronic health record system through external validation against an external clinical data source}},
  booktitle = {BMC Medical Informatics Decis. Mak},
  year = {2019},
  url = {https://openreview.net/forum?id=ECMACzhk81},
}
Figure