Discovering and identifying New York heart association classification from electronic health records

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Rui Zhang, Sisi Ma, Liesa Shanahan, Jessica Munroe, Sarah Horn, Stuart M. SpeedieBmc Medical Informatics Decis. mak 2018 · DOI

TL;DR

This paper tackles the challenge of extracting New York Heart Association (NYHA) functional classification for heart failure patients from electronic health records (EHRs). They found that while NYHA class is crucial for assessing treatment effectiveness, it's poorly documented in structured data but abundant in unstructured clinical notes. Their work demonstrates that Natural Language Processing (NLP) methods, particularly Random Forest with n-gram features, can accurately identify NYHA class from these notes, paving the way for better patient monitoring and treatment evaluation.

What Problem Does This Paper Address?

Heart Failure (HF) is a significant health issue, and Cardiac Resynchronization Therapy (CRT) is a common treatment. A key metric for assessing a patient's response to CRT and the progression of HF is the New York Heart Association (NYHA) functional classification. This system categorizes patients from Class I (no symptoms) to Class IV (severe symptoms). However, consistently tracking a patient's NYHA class over time is difficult because this information is rarely stored in structured fields within Electronic Health Records (EHRs). Instead, it's often embedded within unstructured clinical notes, making it challenging to access and analyze computationally. This lack of accessible, longitudinal NYHA data hinders our ability to understand HF progression, evaluate CRT effectiveness, and personalize patient care.

Methods

The researchers employed a multi-stage approach to extract NYHA classifications from EHRs.

  1. Source Identification: They first explored various data sources within the EHR, including local diagnosis codes, procedure descriptions, and clinical notes, to identify where NYHA class information was documented.
  2. Rule-Based NLP: A rule-based method was developed by iteratively reviewing clinical notes to identify patterns and expressions of NYHA classes. This involved recognizing single digits, Roman numerals, and combinations thereof, with specific attention to patterns indicating co-occurring classes (e.g., "NYHA II/III").
  3. Machine Learning-Based NLP: For a more robust approach, they pre-processed clinical notes by removing stopwords, normalizing terms using lexical variant generation (LVG), and ranking words by term frequency. Two feature sets were created: "bag-of-words" and "n-grams" (ranging from bigrams to five-grams). Three machine learning algorithms—Support Vector Machine (SVM), Logistic Regression, and Random Forest—were then trained and evaluated using these feature sets.
  4. Evaluation: Both the rule-based and machine learning methods were evaluated on a held-out testing dataset using precision, recall, and F-measure metrics, comparing their performance against a manually created reference standard.

Datasets

The study utilized data from Fairview Health Services (FHS), extracted from their EHR and stored in a Clinical Data Repository (CDR) at the University of Minnesota. The dataset comprised EHR data for heart failure patients, including diagnosis codes, procedures, and clinical notes, spanning from December 1, 2011, to January 1, 2016.

A subset of 6174 clinical notes, matched with local diagnosis codes indicating NYHA classification for the same patient and encounter, was used to create a reference standard for training and evaluating the NLP models. This reference standard was split into a training set of 3700 notes and a testing set of 2474 notes.

Key Insights and Ideas

  • Clinical Notes are Goldmines (and Minesweepers): The most striking finding is that while structured EHR fields offer very limited NYHA class information, unstructured clinical notes are a rich source. This highlights the immense, yet largely untapped, potential of clinical narratives for extracting critical patient data.
  • NYHA Documentation is Severely Lacking: The study revealed that only about 19.2% of HF patients had their NYHA class documented at least once. This is significantly lower than the recommended practice of documenting it at every office visit, indicating a widespread gap in clinical documentation.
  • Machine Learning Outperforms Rules: While rule-based methods showed decent performance, machine learning approaches, particularly Random Forest with n-grams, demonstrated superior accuracy in identifying NYHA classes. This suggests that complex patterns and contextual information within notes are better captured by ML models.
  • N-grams are Key for Context: The use of n-gram features (sequences of words) significantly improved the performance of ML models compared to simple bag-of-words. This emphasizes the importance of word order and local context in accurately interpreting clinical text.
  • Class IV is the Toughest: All methods struggled most with identifying NYHA Class IV, likely due to its lower frequency in the dataset. This points to a need for targeted strategies to improve the documentation and extraction of information for more severe HF classifications.

Key Results

  • NYHA Class Documentation Rate: Only 19.2% of HF patients had their NYHA class documented in the EHR. For patients with CRT devices, this rate was higher at 72.9%.
  • Source of NYHA Information: Clinical notes were the richest source, containing NYHA class information for 95.6% of mentions, followed by local diagnosis codes (31.3%) and procedure narratives (2.1%).
  • NLP Performance:
    • The rule-based method achieved an overall F-measure of 93.54%.
  • The best machine learning method, Random Forest with n-gram features, achieved an overall F-measure of 93.78%. This slightly outperformed the rule-based method and significantly outperformed bag-of-words features.
  • Performance for NYHA classes I, II, and III was generally high (over 92% F-measure for the best models), while Class IV had lower recall.

Limitations and Future Work

  • Generalizability of Local Codes: The reliance on local diagnosis codes for creating the reference standard is a limitation, as these codes are specific to the institution and may not be generalizable to other EHR systems.
  • Ambiguous Mentions: The current methods struggle with ambiguous NYHA class mentions (e.g., "NYHA 1 or 2") and cannot differentiate between historical and current NYHA status.
  • Limited Scope of Reference Standard: The reference standard was built by matching clinical notes with local diagnosis codes from the same encounter. Future work could explore a broader range of data sources and potentially manual annotation for a more comprehensive reference.
  • Future Directions: The authors suggest exploring other text-based comment fields within the EHR and developing NLP techniques that can infer NYHA class from related data when it's not explicitly mentioned. Further validation of the machine learning models on diverse EHR data is also recommended.

Summary updated

BibTeX
@article{fiahsi2018discovering,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Rui Zhang and Sisi Ma and Liesa Shanahan and Jessica Munroe and Sarah Horn and Stuart M. Speedie},
  title = {{Discovering and identifying New York heart association classification from electronic health records}},
  journal = {Bmc Medical Informatics Decis. mak},
  year = {2018},
  doi = {10.1186/s12911-018-0625-7},
}
Figure