Comparing emotion feature extraction approaches for predicting depression and anxiety

Hannah Burkhardt, Michael Pullmann, Thomas Hull, Patricia Areán, Trevor CohenNAACL 2022

TL;DR

This paper explores how to best extract emotional signals from text messages to predict depression and anxiety. They found that advanced neural network models (BERT) trained on fine-grained emotions outperform traditional word-counting methods (LIWC), offering more nuanced insights into mental health states.

What Problem Does This Paper Address?

Mental health treatment, particularly talk therapy, relies heavily on assessing patient symptoms over time. While traditional methods involve patient self-reports, analyzing the language patients use in therapy sessions offers a rich, continuous stream of data. The challenge lies in effectively extracting meaningful information about emotional states from this text to accurately gauge depression and anxiety severity. Existing methods, like Linguistic Inquiry and Word Count (LIWC), are useful but have limitations in capturing the nuances of language, such as sarcasm or context-dependent meanings. This paper aims to bridge this gap by comparing the effectiveness of traditional LIWC features with those extracted by more sophisticated BERT-based models trained on a detailed emotion taxonomy.

Methods

The researchers employed two primary methods for extracting emotion features from text:

  • Linguistic Inquiry and Word Count (LIWC): This is a well-established dictionary-based approach that counts words falling into pre-defined categories. They used specific LIWC categories known to correlate with mental health, including first-person pronouns, health, sadness, anxiety, anger, and general positive/negative emotion.
  • BERT-based GoEmotions Classifier: This approach leverages a powerful pre-trained transformer model (BERT) fine-tuned on the GoEmotions dataset, which annotates text with 27 fine-grained emotions (plus a neutral category). This allows for a more nuanced understanding of emotional expression compared to broader LIWC categories.

They then used these extracted features in two ways:

  1. Univariate Mixed-Effects Linear Regression: To understand the association of individual emotion features with symptom severity (PHQ-9 for depression, GAD-7 for anxiety). Mixed-effects models were crucial to account for the fact that multiple messages from the same patient were analyzed.
  2. Random Forest Classifiers: To predict binary depression (MDD) and anxiety (GAD) status. They trained models using LIWC features alone, emotion features alone (from different sources), and combinations of features to assess their predictive power.

Datasets

The study utilized a large corpus of messaging therapy sessions collected from over 6,500 unique patients via the Talkspace platform. This dataset included over 13,000 text documents with corresponding PHQ-9 and GAD-7 scores, totaling over 24 million words. The BERT model was trained on the GoEmotions dataset, a benchmark for fine-grained emotion annotation.

Key Insights and Ideas

  • Nuance Matters: Fine-grained emotion categories, as captured by the BERT-based GoEmotions model, provide more predictive power for depression and anxiety than broader categories from LIWC. This suggests that specific emotional expressions, even those not traditionally linked to mental health, can be highly informative.
  • Complementary Strengths: While BERT-based emotion features excel, LIWC's non-emotion categories (like self-focus and health-related words) still offer valuable, complementary information for predicting mental health status. The best predictive models often combined both types of features.
  • Pride as a Predictor: The study highlighted "pride" as a particularly strong predictor, inversely correlated with depression. This aligns with the idea that depression is often associated with a poor self-image, making expressions of pride more salient.
  • Anxiety vs. Depression Nuances: Different emotions were found to be more strongly associated with either depression or anxiety. For instance, "fear" was a stronger predictor of GAD, while "sadness" was more indicative of MDD.

Key Results

  • Univariate Analysis: Emotion features explained a significant portion of the variance in PHQ-9 and GAD-7 scores, with R² values ranging from 0.770 to 0.788 for PHQ-9 and 0.715 to 0.736 for GAD-7. BERT-based GoEmotions features, particularly for anger and sadness, outperformed LIWC.
  • Predictive Modeling (MDD): Random forest models using the full GoEmotions set combined with LIWC non-emotion features achieved an AUROC of 0.671 for predicting MDD.
  • Predictive Modeling (GAD): The same model combination achieved an AUROC of 0.657 for predicting GAD.
  • Feature Importance: For predicting MDD and GAD, features like "fear," "sadness," and various GoEmotions categories were ranked as most important.

Limitations and Future Work

  • Dataset Bias: The study's data primarily came from young, female, and well-educated participants, and the GoEmotions dataset was trained on Reddit data, which has its own demographic biases. This limits the generalizability of the findings to broader populations.
  • Feature Redundancy: Several emotion features (e.g., sadness from different sources) are highly correlated, though random forests are robust to this.
  • Longitudinal Analysis: The current study focused on cross-sectional associations. Future work could investigate how changes in emotion features over time predict changes in depression and anxiety trajectories.
  • Clinical Deployment: The computational cost of BERT-based models is a consideration for real-time clinical deployment. Future research could explore efficient inference methods or on-device processing.
  • Ethical Considerations: The paper acknowledges the ethical implications of analyzing patient-generated data for mental health inference, emphasizing the need for informed consent and careful consideration of data privacy, especially when commercial entities are involved.

Summary updated

BibTeX
@inproceedings{burkhardt2022comparing,
  author = {Hannah Burkhardt and Michael Pullmann and Thomas Hull and Patricia Areán and Trevor Cohen},
  title = {{Comparing emotion feature extraction approaches for predicting depression and anxiety}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure