CLPsych2019 Shared Task: Predicting Suicide Risk Level from Reddit Posts on Multiple Forums

Victor Ruiz, Lingyun Shi, Wei Quan, Neal Ryan, Candice Biernesser, David Brent, Rich TsuiNAACL 2019

TL;DR

This paper tackles the critical challenge of predicting suicide risk from social media by developing machine learning models that analyze longitudinal Reddit posts. They found that features related to self-referencing, clinical findings, and empathy were most predictive, with a Naïve Bayes model showing promise for identifying users at risk.

What Problem Does This Paper Address?

Suicide remains a significant public health concern, particularly among young people. Social media platforms like Reddit offer a unique, albeit challenging, avenue for individuals to express their thoughts and feelings, including those related to mental health and suicidal ideation. However, leveraging this vast amount of unstructured text data for early detection and intervention is difficult. This paper addresses the need for robust, data-driven methods to identify individuals at risk of suicide by analyzing their online communication patterns on Reddit.

Methods

The researchers employed a multi-faceted approach to feature engineering and predictive modeling. Key techniques include:

  • Natural Language Processing (NLP): Standard NLP pipelines were used for text preprocessing, including sentence splitting, tokenization, stop word removal, part-of-speech tagging, and lemmatization.
  • Feature Engineering: A rich set of features was extracted, categorized into several domains:
  • Clinical Findings: Utilizing the cTAKES system to identify clinical concepts and flagging suicide attempt-related terms.
  • Social Determinants of Health (SDOH): Classifying sentences into 11 predefined social categories.
  • Emotion and Health-Disorder Association: Using lexicons to identify emotion-related words and terms associated with psychological disorders.
  • Readability Scores: Calculating seven different readability metrics to assess text complexity.
  • Semantic Role Labeling (SRL): Employing advanced BiLSTM and ELMo models to identify predicate-argument structures, which can reveal self-referential thinking.
  • Sentiment Levels: Using Stanford CoreNLP to extract sentiment scores (very negative to very positive) and aggregating them per user.
  • Topic Modeling: Applying Latent Dirichlet Allocation (LDA) to identify underlying topics in posts.
  • Empathy Topics: Using a pre-defined empathy lexicon to identify specific themes.
  • Forum Posting Behavior (FPB): Extracting features related to posting habits, such as frequency, timing, and subreddit usage.
  • Doc2Vec: Representing each Reddit post as a 300-dimensional vector.
  • Machine Learning Models: Seven models were developed, including Naïve Bayes (NB), Gradient Boosting (GB), Random Forest (RF), Support Vector Machine (SVM), and deep neural networks (CNN, LSTM). Notably, their augmented deep neural networks incorporated both freetext (Doc2Vec) and user-level aggregate statistics.
  • Oversampling: To address class imbalance (especially for lower risk categories), an oversampling strategy was used.
  • Feature Selection: A two-phase approach involving a correlation-based filter and a forward greedy search was employed.
  • Ensemble Model: The top three performing models were combined into an ensemble classifier.

Datasets

The study utilized two datasets provided by the CLPsych2019 shared task:

  • Training Dataset: 496 Reddit users with 31,553 posts.
  • Test Dataset: 125 Reddit users with 9,610 posts.

Both datasets were annotated with suicide risk levels. The paper notes that the distribution of users across risk categories was similar in both training and test sets, with a notable underrepresentation of "low risk" users.

Key Insights and Ideas

  • Self-Referencing is a Strong Indicator: The most surprising and impactful finding was the strong correlation between self-referential language (e.g., "I," "me," "myself") and negated self-referencing with elevated suicide risk. This aligns with existing literature but highlights its importance in social media analysis.
  • SRL's Power for Self-Reflection: Semantic Role Labeling proved crucial for uncovering self-referential thinking, a key aspect of understanding an individual's internal state.
  • Clinical and Empathy Features Matter: Beyond linguistic patterns, features derived from clinical findings and empathy-related topics were significant predictors.
  • Limitations of General NLP Tools: Standard sentiment analysis tools (trained on movie reviews) and LDA topic modeling showed limited impact, possibly due to the diverse nature of subreddits and the specific context of suicide risk.
  • Oversampling Boosts Performance: The oversampling strategy was effective in improving predictive performance, suggesting its necessity for rare classes.

Key Results

  • The ensemble model achieved the best macro-averaged F1 score of 0.379 on the test dataset, ranking 3rd among participating teams.
  • The SVM model showed strong performance in classifying the four suicide risk categories, likely due to its ability to handle hyperspace parameters and nonlinearity.
  • The Naïve Bayes (NB) model demonstrated the best performance in two binary classification tasks:
  • Flagged risk (any risk other than no risk) vs. no risk: F1 score of 0.836.
  • Urgent risk (moderate or severe risk) vs. others: F1 score of 0.736. The NB model's simplicity and smaller feature set (75 variables) may have contributed to its robust performance.

  • Among the top features, SRL, Empathy, Readability, Clinical findings, and Forum Posting Behavior were most prominent.

Limitations and Future Work

  • Deep Neural Networks Underperformed: The study's deep neural networks (CNN and LSTM) did not perform as well as expected. The authors attribute this to a limited number of users in low/moderate risk categories, a large number of input variables, and suggest exploring more complex DNN structures or multi-DNN approaches to capture temporal dynamics and system non-linearity.
  • Generalizability of Sentiment Tools: The sentiment analysis tool, trained on movie reviews, may not be directly applicable to the nuances of suicide risk prediction.
  • Limited Impact of SDOH and Topic Modeling: The authors suggest that the variety of subreddits might have made it challenging to effectively group topics for suicide risk classification, and the SDOH model's reliance on clinical data might be limited for social media.
  • Future Directions: The paper implies that further research into more sophisticated DNN architectures, domain-specific sentiment analysis, and exploring the interplay of various feature types could lead to improved prediction accuracy. The success of the NB model for binary classification also suggests its potential as a practical tool for identifying users at immediate risk.

Summary updated

BibTeX
@inproceedings{ruiz2019clpsych,
  author = {Victor Ruiz and Lingyun Shi and Wei Quan and Neal Ryan and Candice Biernesser and David Brent and Rich Tsui},
  title = {{CLPsych2019 Shared Task: Predicting Suicide Risk Level from Reddit Posts on Multiple Forums}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure