Similar Minds Post Alike: Assessment of Suicide Risk Using a Hybrid Model

Lushi Chen, Abeer Aldayel, Nikolay Bogoychev, Tao GongNAACL 2019

TL;DR

This paper tackles the critical challenge of automatically assessing suicide risk from online forum posts. The authors propose a novel hybrid model that combines behavioral patterns with language modeling, achieving the best performance in distinguishing different risk levels on the CLPsych 2019 shared task. This work highlights the potential of leveraging diverse user data for more effective suicide risk detection.

What Problem Does This Paper Address?

Suicide is a major global health concern, and early identification of individuals at risk is crucial for prevention. While clinical assessments are vital, they are often time-consuming and may miss individuals who don't explicitly express suicidal thoughts. Online forums, particularly those dedicated to mental health discussions, offer a rich source of data that could potentially reveal subtle indicators of suicide risk. However, accurately interpreting these complex signals and distinguishing between different levels of risk (from "no risk" to "severe risk") remains a significant challenge. This paper addresses this gap by developing an automated system to assess suicide risk based on users' forum postings.

Methods

The authors propose three distinct models and a hybrid approach:

  • Behavioral Model (BM): This model focuses on capturing users' posting behaviors and thoughts. It engineers four groups of features:
  • Posting Behavior: Analyzes patterns like frequency, word count, and timing of posts across various subreddits, including those related to mental health.
  • Sentiment: Assesses a user's overall sentiment profile and their sentiment towards specific topics (e.g., family, self).
  • Content: Utilizes both open (tf-idf) and closed (LIWC, Empath) vocabulary approaches to represent the semantic content of posts.
  • Motivation: Identifies potential motivations for suicide by compiling dictionaries of relevant terms (e.g., financial problems, drug use, mental illness history). A Support Vector Machine (SVM) is used for classification with these features.

  • Suicide Language Model (SLM): This model trains a separate language model for each risk level using all posts from users within that risk category. The likelihood (perplexity) of a user's posts is then calculated for each model, and the model with the lowest perplexity is chosen, indicating the most probable risk level.

  • Hybrid Model (HMBM_SLM): This model combines the strengths of the BM and SLM. It leverages the BM for identifying "no risk" (Class A) and distinguishing between "moderate risk" (Class C) and "severe risk" (Class D), while using the SLM for better identification of "low risk" (Class B). Specific rules are applied to integrate the predictions, such as replacing BM predictions with SLM predictions if the BM's confidence score is below a certain threshold.

Datasets

The study utilizes the dataset provided for the CLPsych 2019 shared task B. This dataset comprises 31,553 posts from 621 users who had posted in the "suicide watch" (SW) subreddit. Users are labeled into four risk levels: "no risk" (Class A), "low risk" (Class B), "moderate risk" (Class C), and "severe risk" (Class D). The paper notes that Class B constitutes a small portion of the dataset (less than 10%), and a significant number of posts lack body content.

Key Insights and Ideas

  • Behavioral cues are vital: The paper strongly suggests that analyzing not just what users say, but how they say it (posting patterns, sentiment, motivations) is crucial for suicide risk assessment.
  • Language models capture nuances: SLMs can effectively model the linguistic characteristics associated with different risk levels, particularly for "low risk" categories where behavioral models might struggle.
  • Hybridization is key: Combining behavioral and language modeling approaches yields superior performance, as the strengths of one model can compensate for the weaknesses of the other.
  • Data imbalance is a challenge: The underrepresentation of "low risk" users in the dataset poses a significant challenge for accurate classification.
  • Irrelevant posts can be informative: The authors hypothesize that even posts not directly related to suicide can provide valuable insights into a user's interaction behavior and emotional state.

Key Results

The hybrid model (HMBM_SLM) achieved the best performance on the CLPsych task B, with an F1 macro average of 38%.

  • BM: Achieved an F1 macro average of 36%, performing consistently on classes A and D but struggling with B and C.
  • SLM: Achieved an F1 macro average of 33%, showing signs of overfitting by classifying too many posts into Class D.
  • HMBM_SLM: Achieved an F1 macro average of 38%, demonstrating improved performance across all classes by strategically combining the strengths of BM and SLM. For instance, it showed better performance in distinguishing flagged vs. non-flagged and urgent vs. non-urgent cases.

Limitations and Future Work

  • Limitations: The authors acknowledge that the SLM model shows signs of overfitting, classifying a disproportionate number of posts into the "severe risk" category. The BM model also struggles with distinguishing between "low" and "moderate" risk levels. The dataset's imbalance, with a low representation of "low risk" users, is also a noted limitation.
  • Future Work: The authors suggest further analysis of individual features to better understand their contribution to suicide risk identification. Exploring more advanced feature engineering and model architectures could also lead to further improvements.

Summary updated

BibTeX
@inproceedings{chen2019similar,
  author = {Lushi Chen and Abeer Aldayel and Nikolay Bogoychev and Tao Gong},
  title = {{Similar Minds Post Alike: Assessment of Suicide Risk Using a Hybrid Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure