Distinguishing Clinical Sentiment: The Importance of Domain Adaptation in Psychiatric Patient Health Records

Eben Holderness, Philip Cawkwell, Kirsten Bolton, James Pustejovsky, Mei-Hua HallNAACL 2019

TL;DR

This paper tackles the challenge of applying sentiment analysis to psychiatric Electronic Health Records (EHRs), finding that standard tools fail to capture clinical nuances. They introduce a novel domain-specific definition of clinical sentiment and an annotation project, demonstrating that even with limited data, a custom model can effectively identify clinical sentiment, paving the way for improved readmission risk prediction.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) contain a wealth of information that could be leveraged to improve patient care. Natural Language Processing (NLP) techniques, particularly sentiment analysis, have shown promise in extracting insights from text. However, sentiment analysis, which typically focuses on general opinions (positive, negative, neutral), has been minimally explored in the clinical domain, especially within psychiatric EHRs. This is problematic because clinical language, particularly in psychiatry, carries specific meanings and nuances that differ significantly from general language. For instance, terms like "depressed" or "hallucinations" have a direct clinical implication that off-the-shelf sentiment tools might misinterpret. The authors highlight a critical gap: the need to adapt sentiment analysis to the unique context of psychiatric EHRs to accurately identify patient sentiment related to specific risk factors, which is crucial for predicting readmission risk and informing treatment.

Methods

The core of this work lies in domain adaptation of sentiment analysis. The authors first define "psychiatric clinical sentiment" as a clinician's attitude (positive, negative, or neutral) towards a patient's prognosis concerning seven identified risk factor domains: appearance, mood, interpersonal relations, substance use, thought content, thought process, and occupation.

They then conducted an annotation project where three clinicians labeled psychiatric EHR texts at the sentence level according to this definition. This resulted in a custom, domain-specific dataset.

For model development, they evaluated three approaches:

  1. Baseline: A majority vote approach using the off-the-shelf "Pattern" sentiment lexicon.
  2. Fully Supervised MLP: A multilayer perceptron trained on the annotated data.
  3. Semi-Supervised MLP (Self-Training): An MLP trained on both the labeled data and a large corpus of unlabeled EHR data.

Crucially, they used the Universal Sentence Encoder (USE) to generate sentence embeddings, which are known to perform well on transfer learning tasks, even when trained on general-domain data. They trained seven separate models for each risk factor domain, acknowledging that sentiment can vary across these domains. To handle the imbalance of neutral examples, they implemented a specific thresholding strategy for classifying neutral sentiment.

Datasets

The paper introduces a newly created psychiatric clinical sentiment dataset derived from EHRs of patients in the OnTrackTM program at McLean Hospital. This dataset was split into:

  • Training Dataset: 3,500 sentences, with examples carefully selected to involve only one risk factor domain.
  • Test Dataset: 1,650 sentences, intentionally designed to be challenging with variable length, vocabulary, and potential for multiple risk factor domains within a single sentence.

They also utilized a large corpus of unannotated psychosis patient EHR data (2,100,000 sentences) for the semi-supervised learning approach.

Key Insights and Ideas

  • Clinical Sentiment is Distinct: Off-the-shelf sentiment analysis tools are inadequate for psychiatric EHRs. They fail to capture the specific meaning and polarity of clinical terms, leading to poor performance.
  • Domain Adaptation is Essential: Creating a domain-specific definition of clinical sentiment and annotating data accordingly is crucial for building effective models.
  • Small Datasets Can Be Powerful: Even with a relatively small amount of labeled data, a custom-trained model can achieve better results than general-purpose tools, highlighting the value of domain-specific training.
  • Risk Factor Specificity Matters: Training separate models for each risk factor domain leads to better performance, as the nuances of sentiment differ across these clinical areas.
  • Neutral Sentiment is Tricky: Identifying neutral sentiment in clinical contexts is particularly challenging due to the inherent bias towards positive or negative documentation and the natural distribution of data.

Key Results

  • The Pattern sentiment lexicon (baseline) showed a strong trend towards underclassifying positive and negative examples, with significantly better performance on neutral examples, confirming its inadequacy for clinical sentiment.
  • The authors' custom-trained fully supervised and semi-supervised models achieved F-measures that exceeded those reported in prior literature for similar tasks, despite the limited training data.
  • The models demonstrated better performance in capturing sentiment related to specific risk factors compared to a general "All" category.
  • Inter-annotator agreement (IAA) was substantial (Pi=0.691-0.768), indicating good reliability in the annotation process.

Limitations and Future Work

Limitations: * Neutral Sentiment Performance: Both fully and semi-supervised models struggled with identifying neutral sentiment across all seven domains, likely due to the inherent bias in clinical documentation and the difficulty in obtaining sufficient neutral examples. * Data Imbalance: The natural distribution of sentiment in EHRs led to an imbalance in training examples, particularly for neutral sentiment.

Future Work: * Increase Annotated Data: The authors plan to expand the annotated training corpus, with a specific focus on increasing neutral examples. * Sequence Models: They intend to move from sentence-level vector representations to sequence models for improved input representation. * Temporal Linking: Future work will incorporate temporal linking of clinical sentiment elements to track changes in patient status over time. * Readmission Risk Prediction: The ultimate goal is to integrate this clinical sentiment model into a larger pipeline for predicting inpatient readmission risk.

Summary updated

BibTeX
@inproceedings{holderness2019distinguishing,
  author = {Eben Holderness and Philip Cawkwell and Kirsten Bolton and James Pustejovsky and Mei-Hua Hall},
  title = {{Distinguishing Clinical Sentiment: The Importance of Domain Adaptation in Psychiatric Patient Health Records}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure