Using natural language processing methods to classify use status of dietary supplements in clinical notes

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Yadan Fan, Rui ZhangBmc Medical Informatics Decis. mak 2018 · DOI

TL;DR

This paper tackles the challenge of automatically extracting dietary supplement usage status from clinical notes, a crucial step for drug-supplement interaction research. They compare rule-based and machine learning approaches, finding that machine learning, particularly Maximum Entropy with a combination of unigrams, bigrams, and indicator words, significantly outperforms rule-based methods on a larger dataset.

What Problem Does This Paper Address?

Dietary supplements are widely used, but their safety is a concern due to potential interactions with prescribed medications, which can lead to dangerous clinical outcomes. Electronic Health Records (EHRs) contain valuable information about supplement use, but this data is often buried within unstructured clinical notes. Manually extracting this information is time-consuming and prone to errors. This paper aims to develop and compare automated methods (rule-based and machine learning) to accurately classify the "use status" (continuing, discontinued, started, unclassified) of dietary supplements mentioned in clinical notes, thereby facilitating large-scale safety research.

Methods

The study employed two main approaches:

  • Rule-Based Classifier: This method relied on identifying specific "indicator words" (e.g., "start," "stop," "continue," "discontinue") and their proximity to supplement mentions within a defined window of tokens. Rules were generated based on these patterns and then evaluated.
  • Machine Learning-Based Classifier: This approach explored various classification algorithms (Decision Tree, Random Forest, Naïve Bayes, SVM, Maximum Entropy) and a range of feature sets. Feature sets included raw unigrams, normalized unigrams, TF-IDF, bigrams, trigrams, and combinations thereof, along with indicator words. The best performing models were identified through 10-fold cross-validation and then evaluated on a held-out test set.

The novelty lies in the systematic comparison of these two approaches on an expanded dataset and the exploration of various feature engineering techniques for the machine learning models.

Datasets

The study utilized a dataset of 2500 sentences extracted from clinical notes at the University of Minnesota, focusing on 25 commonly used dietary supplements. These sentences were manually annotated into four use status classes: Continuing (C), Discontinued (D), Started (S), and Unclassified (U). The dataset was split into a training set (70% of sentences) and a test set (30% of sentences) for evaluation.

Key Insights and Ideas

  • Machine Learning's Scalability Advantage: The study strongly suggests that machine learning models are more efficient and scalable, especially as the dataset size increases. This is a critical finding for real-world applications where large volumes of clinical data need processing.
  • Feature Engineering Matters: The performance of machine learning classifiers was highly dependent on the chosen feature sets. Combinations of unigrams, bigrams, and indicator words proved particularly effective.
  • Indicator Words are Crucial, but Not Sufficient: While indicator words are vital for both rule-based and machine learning approaches, the study highlights that relying solely on them can be limiting, especially when dealing with complex sentence structures or negated statements.
  • The "Unclassified" Category is Tricky: The paper notes that while machine learning improved its performance on the "Unclassified" category compared to previous work, the rule-based classifier still showed higher recall for this category in the current study. This suggests that "unclassified" cases might represent a more nuanced or ambiguous subset of data.

Key Results

  • Machine Learning Outperforms Rule-Based: The optimal machine learning classifier (Maximum Entropy with Type 8 features: unigrams + bigrams + indicators with a window size of 6) achieved an F-measure of 0.90, 0.92, 0.91, and 0.88 for C, D, S, and U statuses, respectively. The rule-based classifier achieved F-measures of 0.90, 0.85, 0.90, and 0.86 for the same statuses.
  • Improved Performance with Larger Dataset: Compared to a previous study with 1300 sentences, the F-measure for the "Discontinued" status significantly improved from 0.85 to 0.92 with the machine learning approach on the 2500-sentence dataset.
  • Individual Supplement Performance: The machine learning classifier demonstrated good generalizability across most of the 25 dietary supplements, with F-measures generally around 0.9. However, it showed poorer performance for Coenzyme Q10 and milk thistle.
  • Error Sources for Rule-Based: The primary sources of error for the rule-based classifier were identified as missing patterns, indicator word issues, and distance issues, accounting for 11.9% of errors.

Limitations and Future Work

  • Limited Feature Sets: The study only explored nine types of features for the machine learning models. Future work could investigate more feature combinations.
  • Deep Learning Potential: The authors suggest exploring state-of-the-art deep learning methods (e.g., Long Short-Term Memory networks) as they can potentially eliminate the need for human-generated features and capture more complex patterns.
  • Hybrid Systems: The paper proposes developing hybrid systems that leverage the strengths of both rule-based (e.g., high sensitivity) and machine learning (e.g., high precision) classifiers to further improve performance.
  • Data Availability: Clinical data containing protected health information could not be shared, which is a common limitation in EHR-based research.

Summary updated

BibTeX
@article{fiahsi2018using,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Yadan Fan and Rui Zhang},
  title = {{Using natural language processing methods to classify use status of dietary supplements in clinical notes}},
  journal = {Bmc Medical Informatics Decis. mak},
  year = {2018},
  doi = {10.1186/s12911-018-0626-6},
}
Figure