Towards Capturing Changes in Mood and Identifying Suicidality Risk

Sravani Boinepelli, Shivansh Subramanian, Abhijeeth Singam, Tathagata Raha, Vasudeva VarmaNAACL 2022

TL;DR

This paper presents novel machine learning and deep learning approaches for identifying shifts in user mood and predicting suicidality risk from social media posts. Their systems achieved top rankings in the CLPsych 2022 Shared Task, demonstrating the potential of analyzing longitudinal user data for mental health insights.

What Problem Does This Paper Address?

Suicide is a significant global health concern, yet social stigma and privacy fears prevent many from seeking professional help. Social media platforms, particularly anonymous ones like Reddit, have become spaces where individuals share their struggles with mental health and suicidal ideation. Identifying critical moments of mood change and assessing suicidality risk from these posts is crucial for timely intervention. However, a key challenge lies in accurately capturing these subtle shifts and risks within the context of a user's evolving online presence, moving beyond simple keyword detection to understanding temporal patterns and individual trajectories.

Methods

The authors tackled two subtasks:

  • Subtask A (Moment of Change - MOC): This involved identifying posts or sequences of posts indicating a shift in a user's mood (IS: switch, IE: escalation, O: other). They employed a Transformer-based LSTM architecture. Sentence transformer models (paraphrase-MiniLM-L6-v2) were used to generate embeddings for each post, which were then fed into an LSTM layer to capture historical context before a final classification. They experimented with different window sizes for the LSTM, finding a window size of 4 to be optimal.
  • Subtask B (Suicidality Risk): This was a user-level classification task to predict risk levels (None, Low, Moderate, Severe) based on Reddit posts from r/SuicideWatch. They explored several approaches:
  • Baseline Models: Logistic Regression, Random Forest, and XGBoost were used with Count Vectors and TF-IDF features.
  • Majority Voting: For user-level prediction, they initially applied majority voting on post-level predictions from their models.
  • Finetuned RoBERTa: They fine-tuned a RoBERTa model for sequence classification, using the first embedding from the model to predict post-level labels, and then applied majority voting for the final user label.
  • Finetuned MOC-LSTM: They leveraged their Subtask A model by treating the task as timeline-level classification. The output of the MOC-LSTM (probability distribution over IS, IE, O) was fed into an additional linear layer and SoftMax to produce a timeline-level probability distribution.

Datasets

The dataset for the CLPsych 2022 Shared Task was used, comprising 255 timelines from users who posted on mental health-related subreddits and r/SuicideWatch. Each timeline contained 10 to 122 posts. The data was derived from multiple sources, including the E-Risk dataset (primarily for Subtask A) and various Reddit datasets (for both subtasks). The dataset was split into training (149 users, 204 timelines, 5143 posts) and testing (36 users, 51 timelines, 1052 posts) sets.

Key Insights and Ideas

  • Temporal Context is Crucial: The paper strongly emphasizes that understanding a user's mood or risk level requires looking beyond individual posts to their historical context. The Transformer-LSTM architecture effectively captures this longitudinal aspect.
  • Post-level vs. User-level: The distinction between identifying moments of change at the post level (Subtask A) and assessing overall risk at the user level (Subtask B) highlights different facets of mental health analysis.
  • Majority Voting as a Simple Yet Effective Strategy: For user-level risk prediction, majority voting on post-level predictions proved to be a strong baseline and a competitive strategy, especially when dealing with limited timeline data.
  • Transformer Models Excel: Finetuned transformer models like RoBERTa demonstrated superior performance, particularly for the more complex user-level risk prediction task, underscoring their ability to learn nuanced patterns in text.
  • Potential for Real-World Application: The authors' focus on real-life applicability and the performance achieved in a shared task setting suggest these methods have practical value for mental health monitoring.

Key Results

The IIITH team achieved strong results in the CLPsych 2022 Shared Task:

  • Subtask A (MOC): They scored a post-level F1 of 0.520 and ranked second with a timeline-level recall of 0.646. Their MOC-LSTM model outperformed baselines by 0.05 in Macro-F1.
  • Subtask B (Suicidality Risk): They ranked second with a micro F1 of 0.520 and third place with a macro F1 of 0.380. The finetuned RoBERTa model scored significantly better than baselines with a 0.08 F1 improvement.

Limitations and Future Work

  • Limited Timeline Span: The authors acknowledge that their majority voting approach for Subtask B might fall short for longer time periods, suggesting the need for more longitudinal and temporal approaches.
  • Technical Issues: The team encountered technical issues that prevented them from officially submitting their finetuned RoBERTa model for Subtask B, despite it showing promising results.
  • Future Directions:
    • Multi-modal Approaches: Integrating data from different social media platforms and modalities (e.g., text, images) could provide a more comprehensive picture of a user's mental health.
  • Longer Timelines: Developing models that can effectively handle and analyze much longer user timelines is a key area for improvement.
  • Real-time, Platform-Agnostic Warnings: The ultimate goal is to build efficient, real-time systems that can provide warnings regardless of the platform.

Summary updated

BibTeX
@inproceedings{boinepelli2022towards,
  author = {Sravani Boinepelli and Shivansh Subramanian and Abhijeeth Singam and Tathagata Raha and Vasudeva Varma},
  title = {{Towards Capturing Changes in Mood and Identifying Suicidality Risk}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure