Automatic mining of symptom severity from psychiatric evaluation notes

George Karystianis, Alejo Nevado‐Holgado, C. Kim, Azad Dehghan, John Keane, Goran NenadićInternational Journal of Methods in Psychiatric Research 2017 · DOI

TL;DR

This paper tackles the challenge of automatically extracting mental health symptom severity from unstructured psychiatric evaluation notes. The authors developed and compared three methods – rule-based, neural network, and hybrid – finding that a rule-based approach achieved the highest accuracy, demonstrating the continued relevance of expert-driven methods in this domain.

What Problem Does This Paper Address?

Electronic Medical Records (EMRs) are increasingly common, offering a wealth of data for clinical decision support and epidemiological research. However, a significant portion of this data, particularly in mental healthcare, exists as free-text narratives. Unlike fields with readily quantifiable metrics (like blood pressure), mental health assessment heavily relies on nuanced descriptions of symptoms and their severity. Manually extracting this information from vast amounts of text is time-consuming and prone to human error. This paper aims to bridge this gap by developing automated methods to accurately determine symptom severity from psychiatric evaluation notes, which is crucial for understanding patient needs and facilitating research.

Methods

The researchers explored three distinct approaches:

  • Rule-Based Approach: This method leverages expert knowledge to create dictionaries of terms and phrases associated with different symptom severity levels (mild, moderate, severe, absent). They then engineered lexical rules based on common syntactic patterns and semantic placeholders to identify these mentions in the text. This approach is knowledge-driven, relying on human expertise to define the extraction logic.
  • Neural Network (NN) Approach: This is a data-driven method that uses a neural network architecture. The NN was designed to integrate various input types, including word stems and regular expression matches, to learn patterns from the data. This approach is more flexible in handling complex linguistic variations but requires substantial labeled data.
  • Hybrid Approach: This method combines the strengths of both the rule-based and NN approaches. It uses the output of the rule-based system as an additional input to the neural network, aiming to improve performance by leveraging both expert knowledge and learned patterns.

Datasets

The study utilized data from the 2016 CEGS N-GRID NLP shared task.

  • Training Set: 325 fully de-identified initial psychiatric evaluation records.
  • Evaluation Set: 216 fully de-identified initial psychiatric evaluation records, unseen during training.

These records were manually annotated for symptom severity by two expert psychiatrists, with a third annotator adjudicating to create a gold standard. The annotation was based on the Research Domain Criteria (RDOC) framework, focusing on the "positive valence" domain of psychiatric symptoms.

Key Insights and Ideas

  • Rule-Based Methods Still Reign: Despite the rise of deep learning, this study found that a carefully crafted rule-based system outperformed neural network approaches for this specific task. This highlights that for domains with limited annotated data, expert knowledge encoded in rules can be highly effective.
  • Data Scarcity is a NN Hurdle: The authors explicitly note that deep NNs struggle with tasks that have limited labeled data, which is a common issue in specialized medical domains like psychiatry. Their NN approaches showed signs of overfitting, indicating difficulty in generalizing to unseen data.
  • Hybridization Challenges: While intuitively appealing, the hybrid approach did not yield better results than the rule-based method alone. This suggests that simply combining outputs might not be sufficient; more sophisticated integration strategies might be needed.
  • Nuance of Psychiatric Notes: The analysis of errors revealed challenges such as dealing with negations, family history mentions, and treatment plans that are not explicitly stated as symptoms, underscoring the complexity of natural language understanding in this context.

Key Results

On the evaluation set:

  • Rule-Based Method: Achieved the highest performance with an MAEM score of 80.1%.
  • Neural Network (NN) Method: Achieved 73.4%.
  • Hybrid Approach: Achieved 72.1%.

The rule-based method demonstrated better generalization, with performance dropping less significantly from training to evaluation compared to the NN methods. The confusion matrix revealed that the biggest confusions were between mild and moderate severity cases, and between severe and moderate cases, indicating that distinguishing between adjacent severity levels is more challenging than differentiating extreme levels.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Training Sample: The rule-based system was developed based on a sample of 50 records, suggesting that a larger set might have allowed for the engineering of more comprehensive rules.
  • Focus on Unstructured Text: The primary focus was on unstructured symptom mentions, potentially overlooking valuable information from structured questions, especially for severe and moderate classes where structured questions were less discriminative.
  • Precedence Rules: The strategy of assigning the highest severity score when multiple symptoms are present might lead to misclassifications.
  • Domain Specificity: The developed methods were trained on initial psychiatric evaluation records and may require adjustments to be applied to other types of mental health records or from different institutions.

Future work could explore:

  • Larger Rule Sets: Engineering more rules with a larger training set.
  • Improved Hybridization: Developing more sophisticated methods for integrating rule-based and NN outputs.
  • Handling Negations and Context: Enhancing the ability to correctly interpret negated statements and contextual information.
  • Combining Severity Signals: Investigating methods to combine different severity signals rather than relying solely on precedence rules.
  • Cross-Domain Applicability: Adapting the models for broader use across different types of mental health records.

Summary updated

Abstract

Although more work is needed to improve the accuracy, the results are encouraging and indicate that automated text mining methods can be used to classify mental health symptom severity from free text psychiatric notes to support epidemiological and clinical research.
BibTeX
@article{karystianis2017automatic,
  author = {George Karystianis and Alejo Nevado‐Holgado and C. Kim and Azad Dehghan and John Keane and Goran Nenadić},
  title = {{Automatic mining of symptom severity from psychiatric evaluation notes}},
  journal = {International Journal of Methods in Psychiatric Research},
  year = {2017},
  doi = {10.1002/mpr.1602},
}
Figure