Emotionally-Informed Models for Detecting Moments of Change and Suicide Risk Levels in Longitudinal Social Media Data

Ulya Bayram, Lamia BenhibaNAACL 2022

TL;DR

This paper introduces novel emotionally-informed models for detecting shifts in users' moods and assessing suicide risk from longitudinal social media data. By leveraging pre-trained language models to capture nuanced emotional states and combining them with sequence-to-sequence autoencoders and advanced machine learning techniques, the authors achieved top performance in a recent shared task, highlighting the potential of these methods for early mental health intervention.

What Problem Does This Paper Address?

Identifying subtle changes in an individual's mental state, particularly those that might indicate a decline in well-being or an increased risk of suicide, is a critical challenge. Social media platforms generate vast amounts of longitudinal data (sequences of posts over time) that could potentially serve as a rich source for such detection. However, analyzing this data presents two main hurdles:

  1. Detecting Mood Changes: Users' emotional states can fluctuate. Identifying not just sudden mood swings (switches) but also gradual progressions (escalations) from a user's timeline of posts is complex. This requires understanding the temporal dynamics of emotions expressed online.
  2. Assessing Suicide Risk: Directly predicting a user's suicide risk level from their social media activity is a sensitive and challenging task. Existing methods often struggle to capture the complex interplay of emotions and temporal patterns that might signal elevated risk.

This work addresses these challenges by developing models that can effectively process longitudinal social media data to pinpoint moments of emotional change and predict suicide risk levels.

Methods

The authors employed a multi-faceted approach, leveraging state-of-the-art natural language processing techniques:

  • Emotionally-Informed Feature Extraction: The core of their approach involves using pre-trained BERT-based models to extract rich emotional and sentiment information from individual posts. Specifically, they used:
  • Bertweet-base-sentiment: For sentiment scores (Positive, Negative, Neutral).
  • EmoRoBERTa: For 28 distinct emotion scores.
  • Twitter-roberta-base-emotion: For 4 emotion scores. These scores were concatenated to form a 35-dimensional "emotionally-informed feature vector" for each post.

  • Mood Anomaly Detection (Task A): To capture mood changes, they used a Sequence-to-Sequence (Seq2Seq) autoencoder. This model was trained to predict the next post's feature vector given a sequence of previous posts. The "abnormality vector" was then computed as the error between the predicted and actual feature vectors, serving as a proxy for mood deviations. This was further augmented with window-based abnormality vectors and the difference between consecutive posts. These features were then fed into machine learning classifiers (Logistic Regression, XGBoost, and an ensemble) to predict mood labels (O for neutral, IE for escalation, IS for switch).

  • Suicide Risk Level Detection (Task B):
    • First Submission: This model combined n-gram features (unigrams and bigrams) with the emotion bandwidth features. Emotion bandwidth was calculated as the absolute difference between the minimum and maximum emotion/sentiment scores across a user's posts. This combined feature set was then used with a Logistic Regression classifier.
  • Second Submission: This approach leveraged the mood change predictions from Task A. A Bi-LSTM (Bidirectional Long Short-Term Memory) network was used to predict suicide risk at the timeline level, considering past and future emotional contexts. Aggregation methods like argmax were used to combine timeline predictions into a user-level risk score.

Datasets

The dataset for this shared task was a compilation of three existing datasets:

  • UMD (2019 CLPsych): From Shing et al. (2018) and Zirikly et al. (2019).
  • E-Risk: With additional data from Losada and Crestani (2016) and Losada et al. (2020).
  • Reddit-New: A new collection from Tsakalidis et al. (2022a).

The dataset comprised 255 timelines, with 204 used for training and 51 for testing. The posts were labeled with mood change indicators (IS, IE, O) and users were assigned suicide risk levels (Low, Moderate, Severe).

Key Insights and Ideas

  • Emotion is Key: The success of their models strongly suggests that capturing nuanced emotional states, beyond simple sentiment, is crucial for understanding mental health signals in social media.
  • Temporal Dynamics Matter: The use of Seq2Seq autoencoders for anomaly detection and Bi-LSTMs for risk prediction highlights the importance of considering the sequence and progression of posts over time, not just individual posts in isolation.
  • Feature Fusion is Powerful: Combining different types of features—linguistic (n-grams), emotional (BERT-derived scores), and temporal (anomaly vectors)—proved highly effective for both tasks.
  • Emotion Bandwidth as a Proxy: The concept of "emotion bandwidth" (the range of emotions expressed by a user) emerged as a surprisingly effective feature for predicting suicide risk.

Key Results

The authors reported strong performance in the 2022 CLPsych shared task:

  • Task A (Mood Change Detection): Their models achieved top-three rankings for macro and micro F1 scores across all evaluation metrics (post-level, timeline coverage, and window-based evaluation). Specifically, the Seq2Seq + XGBoost model showed robust performance, and the Seq2Seq + LR model achieved the third-highest F1 overall and the best macro-average recall on the window-based evaluation.
  • Task B (Suicide Risk Detection): Their first model (n-grams + emotion bandwidth + LR) achieved the highest micro and macro average F1 scores among all participants. The Bi-LSTM model also showed improvement over baselines, though its performance was slightly hampered by aggregation strategies.

Limitations and Future Work

  • Limitations:
    • The Bi-LSTM model for Task B might have benefited from being trained on user-level labels rather than timeline-level labels.
  • The aggregation method (argmax) for the Bi-LSTM model biased predictions towards moderate and severe risk levels, potentially underestimating low-risk cases.
  • The paper mentions that the dataset for Task B excluded users with an "N/A" suicide risk label, which might limit generalizability.
  • Future Work:
    • Exploring different aggregation methods for the Bi-LSTM model to better capture low-level suicide risk.
  • Investigating various look-back time windows for the anomaly detection models to gain deeper insights.
  • Further research into suicide risk detection, especially in scenarios where mood change labels are available or can be reliably obtained.
  • The authors suggest their models could inspire future research for accurately detecting abrupt mood changes and potentially enabling early mental health interventions.

Summary updated

BibTeX
@inproceedings{bayram2022emotionally,
  author = {Ulya Bayram and Lamia Benhiba},
  title = {{Emotionally-Informed Models for Detecting Moments of Change and Suicide Risk Levels in Longitudinal Social Media Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure