TL;DR
This paper presents an informatics pipeline that extracts smoking status, pack year history, and cessation dates from unstructured clinical notes in Electronic Health Records (EHRs). By validating this pipeline against an external questionnaire, the study demonstrates its potential to improve patient identification for smoking cessation and lung cancer screening, despite significant data gaps in current EHRs.
What Problem Does This Paper Address?
Smoking remains a leading cause of preventable illness and death, yet its documentation in EHRs often lacks the detail needed for effective clinical action. While algorithms exist to extract smoking behaviors from clinical notes, their real-world performance and reliability when compared to external, trusted data sources have been poorly understood. This gap hinders the development of robust "smokers' registries" that could proactively identify patients for crucial interventions like smoking cessation counseling and lung cancer screening. The paper aims to bridge this gap by assessing the quality and availability of smoking-related data extracted from EHRs against an external clinical data source.
Methods
The researchers developed and evaluated an informatics pipeline designed to extract three key pieces of smoking-related information from clinical notes:
- Smoking Status: Using a multi-class support vector machine algorithm adapted from previous work, the pipeline classifies notes into categories like "current," "former," "never," and "unknown." This involved identifying "hotspot" words related to smoking and analyzing surrounding text.
- Pack Year History: A regular expression algorithm was employed to parse sentences containing smoking-related terms and extract numerical values representing frequency (packs per day) and duration (years smoked). These were then multiplied to calculate pack years.
- Cessation Date: Sentences containing keywords like "quit" or "stop" were extracted, and any associated dates were converted to a uniform format.
The pipeline was trained and validated on annotated clinical notes. For external validation, the pipeline's output was compared against data from the New Hampshire Colonoscopy Registry (NHCR) questionnaire. The study also explored combining the NLP-extracted data with existing semi-structured EHR fields to assess if this improved performance.
Datasets
- Training and Validation Data: 756 annotated clinical notes from a local Epic EHR data warehouse were used. This included 533 notes for training and 223 for validation.
- External Validation Data: Data from 3,000 participants of the New Hampshire Colonoscopy Registry (NHCR) was used. This registry collects comprehensive colonoscopy data and includes patient questionnaires with smoking status, pack year history, and cessation information. For this study, 1,504 NHCR participants were successfully matched to clinical notes.
Key Insights and Ideas
- Significant Data Gaps in EHRs: A striking finding is that a substantial majority (73%) of the analyzed clinical notes contained no smoking behavior information, highlighting a major challenge for leveraging EHR data for smoking-related interventions.
- NLP as a Complement, Not a Replacement: While NLP can extract valuable information from unstructured notes, it's most effective when combined with existing semi-structured EHR data. This "fact-checking" approach improves reliability.
- External Validation is Crucial: Assessing NLP algorithms against an external, independent data source (like the NHCR questionnaire) is essential for understanding their true clinical utility and performance in real-world scenarios, beyond just comparing against human annotations.
- Lung Cancer Screening as a Viable Use Case: The pipeline demonstrated promise in identifying patients eligible for lung cancer screening, a critical public health initiative. Even with limitations, the ability to flag potential candidates is a significant step.
Key Results
- Data Availability: 73% of the 1,504 matched clinical notes contained no smoking behavior information.
- Smoking Status Concordance: The weighted Cohen's kappa between the external questionnaire and the pipeline-identified smoking status was 0.56 (for pipeline-only data) and 0.62 (for combined pipeline and semi-structured data).
- Pack Year Correlation: The correlation between pack year history reported by the combined EHR sources and the external questionnaire was 0.39 (n=81).
- Lung Cancer Screening Eligibility:
- Combined EHR data (pipeline + semi-structured): Positive Predictive Value (PPV) of 85.4%, Negative Predictive Value (NPV) of 83.8%, Sensitivity of 63.1%, and Specificity of 94.7%.
- Pipeline-only data: Showed a significant drop in sensitivity (20.9%) compared to the combined approach.
Limitations and Future Work
- Data Missingness: The most significant limitation is the high rate of missing smoking behavior information in EHRs.
- Pack Year Underestimation: The pipeline sometimes underestimated pack year history due to misclassifying frequency or duration as full pack years. The authors suggest heuristic cutoffs or further algorithm refinement.
- Cessation Date Limitations: The external questionnaire did not capture cessation dates, preventing a full validation of that pipeline component.
- Cross-sectional Design: The study used a cross-sectional design; future work should explore longitudinal data.
- Generalizability: The use of a cancer screening questionnaire for external validation might introduce bias, and the patient population may not be fully representative of all clinical settings.
- Future Directions: The authors propose external validation on different hospital systems, whole-record classification to reduce missing data, and developing the software for wider integration into EHRs and for public release. They also emphasize the need to identify more use cases and develop comprehensive reporting for clinicians.
Summary updated