Suicide Risk Assessment with Multi-level Dual-Context Language and BERT

Matthew Matero, Akash Idnani, Youngseo Son, Salvatore Giorgi, Huy Vu, Mohammad Zamani, Parth Limbachiya, Sharath Chandra Guntuku, H. Andrew SchwartzNAACL 2019

TL;DR

This paper introduces a novel "dual-context" approach to suicide risk assessment using language from online forums. By treating suicide-specific content separately from general user posts and incorporating both theoretical linguistic features and powerful BERT embeddings, they achieve state-of-the-art results in predicting suicide risk levels.

What Problem Does This Paper Address?

Accurately assessing suicide risk is a critical challenge, and researchers are increasingly turning to automated analysis of user-generated content for early detection. Existing methods often treat all text from a user as a single, undifferentiated context and may focus on only one level of analysis (e.g., individual messages or overall user behavior). This paper addresses the limitations of these approaches by proposing a more nuanced model that considers the distinct nature of suicide-related discussions versus general online interactions, and integrates multiple levels of linguistic and user-level information.

Methods

The core innovation is the dual-context modeling approach. This involves processing language from suicide-specific forums (like r/SuicideWatch) separately from a user's other online posts. They explore two main types of models:

  • Traditional ML Models: Using regularized logistic regression with a combination of features.
  • Novel Deep Learning Architecture (DualDeepAtt): This architecture employs two separate GRU cells, one for suicide-context posts and another for non-suicide-context posts. It then uses attention mechanisms to combine information from both contexts and incorporates a "user-factor adaptation" layer, which integrates user-level traits like age, gender, and inferred latent factors.

They also investigate various feature sets:

  • Theoretical Dimensions: Linguistic features capturing emotional valence, arousal, personality traits (e.g., neuroticism, agreeableness), and demographic inferences.
  • Open-Vocabulary Features: Dimensionally reduced BERT embeddings and LDA topics to capture the nuances of language.
  • Meta-features: User-level post statistics and subreddit information.

Datasets

The primary dataset used is from the CLPsych 2019 Shared Task, collected from Reddit. This dataset includes user posts annotated into four suicide risk categories: no risk, low, moderate, and severe. The task was sub-divided into three parts (Task A, B, and C) based on the data available (e.g., only suicide-related posts, or all posts including control users).

Key Insights and Ideas

  • Context Matters: Treating suicide-related content separately from general user posts significantly improves prediction accuracy. This highlights that the nature of the discussion is crucial, not just the volume of text.
  • Different Features for Different Risks: Affective features from suicide contexts are good at distinguishing between "any risk" and "no risk," but personality factors from non-suicide contexts are better at differentiating between the levels of risk (low, medium, high). This suggests a complex interplay between different types of language and risk.
  • BERT Embeddings are Powerful: BERT embeddings, even when dimensionally reduced, provide a strong foundation for suicide risk prediction, especially when combined with other features.
  • User-Level Traits Add Value: Incorporating user-level information like personality and inferred latent factors can enhance model performance, suggesting that individual predispositions play a role.

Key Results

The paper reports state-of-the-art performance on the CLPsych 2019 Shared Task (Task B). Their dual-context approach, specifically the DualContextBert model (a logistic regression model using BERT embeddings from both contexts), achieved an F1 score of 0.50 on the hidden test set. This was a significant improvement over single-context models. For Task A, the DeepAtt neural model also performed well, achieving an F1 score of 0.50.

Limitations and Future Work

  • Task C Difficulty: Task C, which involved predicting risk for users who were mostly "no risk" and evaluated only on those with some risk, proved to be the most challenging. The dual-context approach was not directly applicable here.
  • Limited Training Data: The authors note that their novel neural architecture (DualDeepAtt) might have been "hindered by the limited amount of training, relative to parameters for the task."
  • Future Directions: The paper suggests that combining BERT-based features with theoretical dimensions is a promising avenue for building more accurate predictive models. Further exploration of aggregation techniques and user-factor adaptation could also yield improvements.

Summary updated

BibTeX
@inproceedings{matero2019suicide,
  author = {Matthew Matero and Akash Idnani and Youngseo Son and Salvatore Giorgi and Huy Vu and Mohammad Zamani and Parth Limbachiya and Sharath Chandra Guntuku and H. Andrew Schwartz},
  title = {{Suicide Risk Assessment with Multi-level Dual-Context Language and BERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure